 MDEV-18644: Support full_crc32 for page_compressed
This is a follow-up task to MDEV-12026, which introduced
innodb_checksum_algorithm=full_crc32 and a simpler page format.
MDEV-12026 did not enable full_crc32 for page_compressed tables,
which we will be doing now.
This is joint work with Thirunarayanan Balathandayuthapani.
For innodb_checksum_algorithm=full_crc32 we change the
page_compressed format as follows:
FIL_PAGE_TYPE: The most significant bit will be set to indicate
page_compressed format. The least significant bits will contain
the compressed page size, rounded up to a multiple of 256 bytes.
The checksum will be stored in the last 4 bytes of the page
(whether it is the full page or a page_compressed page whose
size is determined by FIL_PAGE_TYPE), covering all preceding
bytes of the page. If encryption is used, then the page will
be encrypted between compression and computing the checksum.
For page_compressed, FIL_PAGE_LSN will not be repeated at
the end of the page.
FSP_SPACE_FLAGS (already implemented as part of MDEV-12026):
We will store the innodb_compression_algorithm that may be used
to compress pages. Previously, the choice of algorithm was written
to each compressed data page separately, and one would be unable
to know in advance which compression algorithm(s) are used.
fil_space_t::full_crc32_page_compressed_len(): Determine if the
page_compressed algorithm of the tablespace needs to know the
exact length of the compressed data. If yes, we will reserve and
write an extra byte for this right before the checksum.
buf_page_is_compressed(): Determine if a page uses page_compressed
(in any innodb_checksum_algorithm).
fil_page_decompress(): Pass also fil_space_t::flags so that the
format can be determined.
buf_page_is_zeroes(): Check if a page is full of zero bytes.
buf_page_full_crc32_is_corrupted(): Renamed from
buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32,
we always simply validate the checksum to the page contents,
while the physical page size is explicitly specified by an
unencrypted part of the page header.
buf_page_full_crc32_size(): Determine the size of a full_crc32 page.
buf_dblwr_check_page_lsn(): Make this a debug-only function, because
it involves potentially costly lookups of fil_space_t.
create_table_info_t::check_table_options(),
ha_innobase::check_if_supported_inplace_alter(): Do allow the creation
of SPATIAL INDEX with full_crc32 also when page_compressed is used.
commit_cache_norebuild(): Preserve the compression algorithm when
updating the page_compression_level.
dict_tf_to_fsp_flags(): Set the flags for page compression algorithm.
FIXME: Maybe there should be a table option page_compression_algorithm
and a session variable to back it?
7 years ago MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.
Allow instant DROP COLUMN and instant changing the order of columns.
The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.
Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.
The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.
Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.
For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.
If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.
Limitations (future work):
MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large
btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.
dict_col_t::DROPPED: Magic value for dict_col_t::ind.
dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".
dict_col_t::is_added(): Rename from dict_col_t::is_instant().
dtype_t::metadata_blob_init(): Initialize the mblob data type.
dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.
dict_table_t::instant: Metadata about dropped or reordered columns.
dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.
dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.
dict_table_t::find(): Find an old column based on a new column number.
dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.
dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().
dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().
dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.
ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)
instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.
ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.
innobase_build_col_map(): Skip added virtual columns.
prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.
commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().
innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.
innodb_update_cols(): Renamed from innodb_update_n_cols().
innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.
innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().
innobase_instant_add_col(): Replace the parameter dfield with type.
innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.
innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().
innobase_rename_column_try(): Skip dropped columns.
commit_cache_norebuild(): Update table->fts->doc_col.
dict_mem_table_col_rename_low(): Skip dropped columns.
trx_undo_rec_get_partial_row(): Skip dropped columns.
trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.
trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.
row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.
row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.
row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.
row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).
row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.
row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.
btr_index_rec_validate(): Recognize the metadata record.
btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.
btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.
btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.
btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.
btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.
btr_cur_trim(): Handle the mblob record.
row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.
btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.
btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.
btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.
REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.
REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.
rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.
rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.
rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).
rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.
rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.
rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.
rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.
rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.
row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.
row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.
row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.
Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
7 years ago MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.
Allow instant DROP COLUMN and instant changing the order of columns.
The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.
Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.
The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.
Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.
For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.
If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.
Limitations (future work):
MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large
btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.
dict_col_t::DROPPED: Magic value for dict_col_t::ind.
dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".
dict_col_t::is_added(): Rename from dict_col_t::is_instant().
dtype_t::metadata_blob_init(): Initialize the mblob data type.
dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.
dict_table_t::instant: Metadata about dropped or reordered columns.
dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.
dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.
dict_table_t::find(): Find an old column based on a new column number.
dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.
dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().
dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().
dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.
ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)
instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.
ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.
innobase_build_col_map(): Skip added virtual columns.
prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.
commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().
innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.
innodb_update_cols(): Renamed from innodb_update_n_cols().
innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.
innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().
innobase_instant_add_col(): Replace the parameter dfield with type.
innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.
innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().
innobase_rename_column_try(): Skip dropped columns.
commit_cache_norebuild(): Update table->fts->doc_col.
dict_mem_table_col_rename_low(): Skip dropped columns.
trx_undo_rec_get_partial_row(): Skip dropped columns.
trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.
trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.
row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.
row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.
row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.
row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).
row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.
row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.
btr_index_rec_validate(): Recognize the metadata record.
btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.
btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.
btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.
btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.
btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.
btr_cur_trim(): Handle the mblob record.
row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.
btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.
btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.
btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.
REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.
REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.
rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.
rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.
rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).
rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.
rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.
rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.
rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.
rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.
row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.
row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.
row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.
Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
7 years ago MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.
Allow instant DROP COLUMN and instant changing the order of columns.
The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.
Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.
The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.
Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.
For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.
If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.
Limitations (future work):
MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large
btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.
dict_col_t::DROPPED: Magic value for dict_col_t::ind.
dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".
dict_col_t::is_added(): Rename from dict_col_t::is_instant().
dtype_t::metadata_blob_init(): Initialize the mblob data type.
dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.
dict_table_t::instant: Metadata about dropped or reordered columns.
dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.
dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.
dict_table_t::find(): Find an old column based on a new column number.
dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.
dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().
dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().
dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.
ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)
instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.
ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.
innobase_build_col_map(): Skip added virtual columns.
prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.
commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().
innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.
innodb_update_cols(): Renamed from innodb_update_n_cols().
innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.
innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().
innobase_instant_add_col(): Replace the parameter dfield with type.
innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.
innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().
innobase_rename_column_try(): Skip dropped columns.
commit_cache_norebuild(): Update table->fts->doc_col.
dict_mem_table_col_rename_low(): Skip dropped columns.
trx_undo_rec_get_partial_row(): Skip dropped columns.
trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.
trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.
row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.
row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.
row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.
row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).
row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.
row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.
btr_index_rec_validate(): Recognize the metadata record.
btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.
btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.
btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.
btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.
btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.
btr_cur_trim(): Handle the mblob record.
row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.
btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.
btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.
btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.
REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.
REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.
rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.
rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.
rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).
rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.
rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.
rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.
rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.
rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.
row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.
row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.
row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.
Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
7 years ago  MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-12266: Change dict_table_t::space to fil_space_t*
InnoDB always keeps all tablespaces in the fil_system cache.
The fil_system.LRU is only for closing file handles; the
fil_space_t and fil_node_t for all data files will remain
in main memory. Between startup to shutdown, they can only be
created and removed by DDL statements. Therefore, we can
let dict_table_t::space point directly to the fil_space_t.
dict_table_t::space_id: A numeric tablespace ID for the corner cases
where we do not have a tablespace. The most prominent examples are
ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file.
There are a few functional differences; most notably:
(1) DROP TABLE will delete matching .ibd and .cfg files,
even if they were not attached to the data dictionary.
(2) Some error messages will report file names instead of numeric IDs.
There still are many functions that use numeric tablespace IDs instead
of fil_space_t*, and many functions could be converted to fil_space_t
member functions. Also, Tablespace and Datafile should be merged with
fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use
fil_space_t& instead of a numeric ID, and after moving to a single
buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to
fil_space_t::page_hash.
FilSpace: Remove. Only few calls to fil_space_acquire() will remain,
and gradually they should be removed.
mtr_t::set_named_space_id(ulint): Renamed from set_named_space(),
to prevent accidental calls to this slower function. Very few
callers remain.
fseg_create(), fsp_reserve_free_extents(): Take fil_space_t*
as a parameter instead of a space_id.
fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(),
fil_name_write_rename(), fil_rename_tablespace(). Mariabackup
passes the parameter log=false; InnoDB passes log=true.
dict_mem_table_create(): Take fil_space_t* instead of space_id
as parameter.
dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter
'status' with 'bool cached'.
dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name.
fil_ibd_open(): Return the tablespace.
fil_space_t::set_imported(): Replaces fil_space_set_imported().
truncate_t: Change many member function parameters to fil_space_t*,
and remove page_size parameters.
row_truncate_prepare(): Merge to its only caller.
row_drop_table_from_cache(): Assert that the table is persistent.
dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL
if the tablespace has been discarded.
row_import_update_discarded_flag(): Remove a constant parameter.
8 years ago MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.
This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.
ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.
Some usability limitations will be addressed in subsequent work:
MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE
The format of the clustered index (PRIMARY KEY) is changed as follows:
(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.
(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.
(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.
(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)
We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.
This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.
The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.
Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.
When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.
UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.
len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.
dict_col_t::def_val: The 'default row' value of the column. If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.
dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().
dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.
dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().
dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.
dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).
dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.
dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().
dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.
dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.
dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().
dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.
dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.
prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.
dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.
dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).
dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().
pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().
row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().
create_index_dict(): Replaces row_merge_create_index_graph().
innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.
btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.
dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.
dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.
dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().
row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.
commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)
PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.
page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.
page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.
page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.
page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.
rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.
rec_offs_make_valid(): Add the parameter 'leaf'.
rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.
dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().
rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.
cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.
trx_t::in_rollback: Make the field available in non-debug builds.
trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.
btr_index_rec_validate(): Account for instant ADD COLUMN
row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.
row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.
dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.
btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.
row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().
rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.
rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.
btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.
row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.
rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.
REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.
rec_comp_status_t: An enum of the status bit values.
rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
8 years ago MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.
This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.
ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.
Some usability limitations will be addressed in subsequent work:
MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE
The format of the clustered index (PRIMARY KEY) is changed as follows:
(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.
(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.
(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.
(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)
We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.
This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.
The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.
Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.
When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.
UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.
len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.
dict_col_t::def_val: The 'default row' value of the column. If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.
dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().
dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.
dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().
dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.
dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).
dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.
dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().
dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.
dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.
dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().
dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.
dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.
prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.
dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.
dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).
dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().
pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().
row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().
create_index_dict(): Replaces row_merge_create_index_graph().
innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.
btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.
dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.
dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.
dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().
row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.
commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)
PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.
page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.
page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.
page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.
page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.
rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.
rec_offs_make_valid(): Add the parameter 'leaf'.
rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.
dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().
rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.
cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.
trx_t::in_rollback: Make the field available in non-debug builds.
trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.
btr_index_rec_validate(): Account for instant ADD COLUMN
row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.
row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.
dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.
btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.
row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().
rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.
rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.
btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.
row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.
rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.
REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.
rec_comp_status_t: An enum of the status bit values.
rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
8 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.
This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.
ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.
Some usability limitations will be addressed in subsequent work:
MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE
The format of the clustered index (PRIMARY KEY) is changed as follows:
(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.
(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.
(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.
(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)
We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.
This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.
The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.
Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.
When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.
UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.
len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.
dict_col_t::def_val: The 'default row' value of the column. If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.
dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().
dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.
dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().
dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.
dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).
dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.
dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().
dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.
dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.
dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().
dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.
dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.
prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.
dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.
dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).
dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().
pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().
row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().
create_index_dict(): Replaces row_merge_create_index_graph().
innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.
btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.
dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.
dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.
dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().
row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.
commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)
PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.
page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.
page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.
page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.
page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.
rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.
rec_offs_make_valid(): Add the parameter 'leaf'.
rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.
dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().
rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.
cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.
trx_t::in_rollback: Make the field available in non-debug builds.
trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.
btr_index_rec_validate(): Account for instant ADD COLUMN
row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.
row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.
dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.
btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.
row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().
rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.
rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.
btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.
row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.
rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.
REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.
rec_comp_status_t: An enum of the status bit values.
rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
8 years ago MDEV-15662 Instant DROP COLUMN or changing the order of columns
Allow ADD COLUMN anywhere in a table, not only adding as the
last column.
Allow instant DROP COLUMN and instant changing the order of columns.
The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.
Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.
The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.
Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.
For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.
If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.
Limitations (future work):
MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large
btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.
dict_col_t::DROPPED: Magic value for dict_col_t::ind.
dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".
dict_col_t::is_added(): Rename from dict_col_t::is_instant().
dtype_t::metadata_blob_init(): Initialize the mblob data type.
dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.
dict_table_t::instant: Metadata about dropped or reordered columns.
dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.
dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.
dict_table_t::find(): Find an old column based on a new column number.
dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.
dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().
dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().
dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.
ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)
instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.
ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.
innobase_build_col_map(): Skip added virtual columns.
prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.
commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().
innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.
innodb_update_cols(): Renamed from innodb_update_n_cols().
innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.
innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().
innobase_instant_add_col(): Replace the parameter dfield with type.
innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.
innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().
innobase_rename_column_try(): Skip dropped columns.
commit_cache_norebuild(): Update table->fts->doc_col.
dict_mem_table_col_rename_low(): Skip dropped columns.
trx_undo_rec_get_partial_row(): Skip dropped columns.
trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.
trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.
row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.
row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.
row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.
row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).
row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.
row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.
btr_index_rec_validate(): Recognize the metadata record.
btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.
btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.
btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.
btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.
btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.
btr_cur_trim(): Handle the mblob record.
row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.
btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.
btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.
btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.
REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.
REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.
rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.
rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.
rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).
rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.
rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.
rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.
rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.
rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.
row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.
row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.
row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.
Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.
7 years ago MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.
This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.
ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.
Some usability limitations will be addressed in subsequent work:
MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE
The format of the clustered index (PRIMARY KEY) is changed as follows:
(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.
(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.
(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.
(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)
We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.
This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.
The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.
Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.
When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.
UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.
len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.
dict_col_t::def_val: The 'default row' value of the column. If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.
dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().
dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.
dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().
dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.
dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).
dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.
dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().
dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.
dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.
dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().
dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.
dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.
prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.
dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.
dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).
dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().
pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().
row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().
create_index_dict(): Replaces row_merge_create_index_graph().
innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.
btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.
dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.
dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.
dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().
row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.
commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)
PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.
page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.
page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.
page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.
page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.
rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.
rec_offs_make_valid(): Add the parameter 'leaf'.
rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.
dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().
rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.
cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.
trx_t::in_rollback: Make the field available in non-debug builds.
trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.
btr_index_rec_validate(): Account for instant ADD COLUMN
row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.
row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.
dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.
btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.
row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().
rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.
rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.
btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.
btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.
row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.
rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.
REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.
rec_comp_status_t: An enum of the status bit values.
rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
8 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed.
[TODO: It appears that the resetting is not taking place as often as
it could be. We should test that a simple INSERT should eventually
cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is
invoked soon enough.]
The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR
are used by multi-versioning. After the history is no longer needed, these
columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert).
When a reader sees 0 in the DB_TRX_ID column, it can instantly determine
that the record is present the read view. There is no need to acquire
the transaction system mutex to check if the transaction exists, because
writes can never be conducted by a transaction whose ID is 0.
The persistent InnoDB undo log used to be split into two parts:
insert_undo and update_undo. The insert_undo log was discarded at
transaction commit or rollback, and the update_undo log was processed
by the purge subsystem. As part of this change, we will only generate
a single undo log for new transactions, and the purge subsystem will
reset the DB_TRX_ID whenever a clustered index record is touched.
That is, all persistent undo log will be preserved at transaction commit
or rollback, to be removed by purge.
The InnoDB redo log format is changed in two ways:
We remove the redo log record type MLOG_UNDO_HDR_REUSE, and
we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the
DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table.
This is also changing the format of persistent InnoDB data files:
undo log and clustered index leaf page records. It will still be
possible via import and export to exchange data files with earlier
versions of MariaDB. The change to clustered index leaf page records
is simple: we allow DB_TRX_ID to be 0.
When it comes to the undo log, we must be able to upgrade from earlier
MariaDB versions after a clean shutdown (no redo log to apply).
While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0)
before an upgrade, to empty the undo logs, we cannot assume that this
has been done. So, separate insert_undo log may exist for recovered
uncommitted transactions. These transactions may be automatically
rolled back, or they may be in XA PREPARE state, in which case InnoDB
will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK.
Upgrade has been tested by starting up MariaDB 10.2 with
./mysql-test-run --manual-gdb innodb.read_only_recovery
and then starting up this patched server with
and without --innodb-read-only.
trx_undo_ptr_t::undo: Renamed from update_undo.
trx_undo_ptr_t::old_insert: Renamed from insert_undo.
trx_rseg_t::undo_list: Renamed from update_undo_list.
trx_rseg_t::undo_cached: Merged from update_undo_cached
and insert_undo_cached.
trx_rseg_t::old_insert_list: Renamed from insert_undo_list.
row_purge_reset_trx_id(): New function to reset the columns.
This will be called for all undo processing in purge
that does not remove the clustered index record.
trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the
old DB_TRX_ID of the record to the undo log.
ReadView::changes_visible(): Allow id==0. (Return true for it.
This is what speeds up the MVCC.)
row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read():
Implement a fast path for DB_TRX_ID=0.
Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type.
MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format!
innobase_start_or_create_for_mysql(): Set srv_undo_sources before
starting any transactions.
The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully
tested by running the following:
./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680
grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err
8 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.
a lot of functions: replace ulint with offset_t
btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
allocate record offsets on the stack instead of waiting for rec_get_offsets()
to allocate it from mem_heap_t. So, reducing memory allocations.
RECORD_OFFSET, INDEX_OFFSET:
now it's less convenient to store pointers in offset_t*
array. One pointer occupies now several offset_t. And those constant are start
indexes into array to places where to store pointer values
REC_OFFS_HEADER_SIZE: adjusted for the new reality
REC_OFFS_NORMAL_SIZE:
increase size from 100 to 300 which means less heap allocations.
And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
is smaller than previous 800 bytes.
REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality
rem0rec.h, rem0rec.ic, rem0rec.cc:
various arguments, return values and local variables types were changed to
fix numerous integer conversions issues.
enum field_type_t:
offset types concept was introduces which replaces old offset flags stuff.
Like in earlier version, 2 upper bits are used to store offset type.
And this enum represents those types.
REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed
get_type(), set_type(), get_value(), combine():
these are convenience functions to work with offsets and it's types
rec_offs_base()[0]:
still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL
rec_offs_base()[i]:
these have type offset_t now. Two upper bits contains type.
6 years ago MDEV-6076 Persistent AUTO_INCREMENT for InnoDB
This should be functionally equivalent to WL#6204 in MySQL 8.0.0, with
the notable difference that the file format changes are limited to
repurposing a previously unused data field in B-tree pages.
For persistent InnoDB tables, write the last used AUTO_INCREMENT
value to the root page of the clustered index, in the previously
unused (0) PAGE_MAX_TRX_ID field, now aliased as PAGE_ROOT_AUTO_INC.
Unlike some other previously unused InnoDB data fields, this one was
actually always zero-initialized, at least since MySQL 3.23.49.
The writes to PAGE_ROOT_AUTO_INC are protected by SX or X latch on the
root page. The SX latch will allow concurrent read access to the root
page. (The field PAGE_ROOT_AUTO_INC will only be read on the
first-time call to ha_innobase::open() from the SQL layer. The
PAGE_ROOT_AUTO_INC can only be updated when executing SQL, so
read/write races are not possible.)
During INSERT, the PAGE_ROOT_AUTO_INC is updated by the low-level
function btr_cur_search_to_nth_level(), adding no extra page
access. [Adaptive hash index lookup will be disabled during INSERT.]
If some rare UPDATE modifies an AUTO_INCREMENT column, the
PAGE_ROOT_AUTO_INC will be adjusted in a separate mini-transaction in
ha_innobase::update_row().
When a page is reorganized, we have to preserve the PAGE_ROOT_AUTO_INC
field.
During ALTER TABLE, the initial AUTO_INCREMENT value will be copied
from the table. ALGORITHM=COPY and online log apply in LOCK=NONE will
update PAGE_ROOT_AUTO_INC in real time.
innodb_col_no(): Determine the dict_table_t::cols[] element index
corresponding to a Field of a non-virtual column.
(The MySQL 5.7 implementation of virtual columns breaks the 1:1
relationship between Field::field_index and dict_table_t::cols[].
Virtual columns are omitted from dict_table_t::cols[]. Therefore,
we must translate the field_index of AUTO_INCREMENT columns into
an index of dict_table_t::cols[].)
Upgrade from old data files:
By default, the AUTO_INCREMENT sequence in old data files would appear
to be reset, because PAGE_MAX_TRX_ID or PAGE_ROOT_AUTO_INC would contain
the value 0 in each clustered index page. In new data files,
PAGE_ROOT_AUTO_INC can only be 0 if the table is empty or does not contain
any AUTO_INCREMENT column.
For backward compatibility, we use the old method of
SELECT MAX(auto_increment_column) for initializing the sequence.
btr_read_autoinc(): Read the AUTO_INCREMENT sequence from a new-format
data file.
btr_read_autoinc_with_fallback(): A variant of btr_read_autoinc()
that will resort to reading MAX(auto_increment_column) for data files
that did not use AUTO_INCREMENT yet. It was manually tested that during
the execution of innodb.autoinc_persist the compatibility logic is
not activated (for new files, PAGE_ROOT_AUTO_INC is never 0 in nonempty
clustered index root pages).
initialize_auto_increment(): Replaces
ha_innobase::innobase_initialize_autoinc(). This initializes
the AUTO_INCREMENT metadata. Only called from ha_innobase::open().
ha_innobase::info_low(): Do not try to lazily initialize
dict_table_t::autoinc. It must already have been initialized by
ha_innobase::open() or ha_innobase::create().
Note: The adjustments to class ha_innopart were not tested, because
the source code (native InnoDB partitioning) is not being compiled.
9 years ago MDEV-6076 Persistent AUTO_INCREMENT for InnoDB
This should be functionally equivalent to WL#6204 in MySQL 8.0.0, with
the notable difference that the file format changes are limited to
repurposing a previously unused data field in B-tree pages.
For persistent InnoDB tables, write the last used AUTO_INCREMENT
value to the root page of the clustered index, in the previously
unused (0) PAGE_MAX_TRX_ID field, now aliased as PAGE_ROOT_AUTO_INC.
Unlike some other previously unused InnoDB data fields, this one was
actually always zero-initialized, at least since MySQL 3.23.49.
The writes to PAGE_ROOT_AUTO_INC are protected by SX or X latch on the
root page. The SX latch will allow concurrent read access to the root
page. (The field PAGE_ROOT_AUTO_INC will only be read on the
first-time call to ha_innobase::open() from the SQL layer. The
PAGE_ROOT_AUTO_INC can only be updated when executing SQL, so
read/write races are not possible.)
During INSERT, the PAGE_ROOT_AUTO_INC is updated by the low-level
function btr_cur_search_to_nth_level(), adding no extra page
access. [Adaptive hash index lookup will be disabled during INSERT.]
If some rare UPDATE modifies an AUTO_INCREMENT column, the
PAGE_ROOT_AUTO_INC will be adjusted in a separate mini-transaction in
ha_innobase::update_row().
When a page is reorganized, we have to preserve the PAGE_ROOT_AUTO_INC
field.
During ALTER TABLE, the initial AUTO_INCREMENT value will be copied
from the table. ALGORITHM=COPY and online log apply in LOCK=NONE will
update PAGE_ROOT_AUTO_INC in real time.
innodb_col_no(): Determine the dict_table_t::cols[] element index
corresponding to a Field of a non-virtual column.
(The MySQL 5.7 implementation of virtual columns breaks the 1:1
relationship between Field::field_index and dict_table_t::cols[].
Virtual columns are omitted from dict_table_t::cols[]. Therefore,
we must translate the field_index of AUTO_INCREMENT columns into
an index of dict_table_t::cols[].)
Upgrade from old data files:
By default, the AUTO_INCREMENT sequence in old data files would appear
to be reset, because PAGE_MAX_TRX_ID or PAGE_ROOT_AUTO_INC would contain
the value 0 in each clustered index page. In new data files,
PAGE_ROOT_AUTO_INC can only be 0 if the table is empty or does not contain
any AUTO_INCREMENT column.
For backward compatibility, we use the old method of
SELECT MAX(auto_increment_column) for initializing the sequence.
btr_read_autoinc(): Read the AUTO_INCREMENT sequence from a new-format
data file.
btr_read_autoinc_with_fallback(): A variant of btr_read_autoinc()
that will resort to reading MAX(auto_increment_column) for data files
that did not use AUTO_INCREMENT yet. It was manually tested that during
the execution of innodb.autoinc_persist the compatibility logic is
not activated (for new files, PAGE_ROOT_AUTO_INC is never 0 in nonempty
clustered index root pages).
initialize_auto_increment(): Replaces
ha_innobase::innobase_initialize_autoinc(). This initializes
the AUTO_INCREMENT metadata. Only called from ha_innobase::open().
ha_innobase::info_low(): Do not try to lazily initialize
dict_table_t::autoinc. It must already have been initialized by
ha_innobase::open() or ha_innobase::create().
Note: The adjustments to class ha_innopart were not tested, because
the source code (native InnoDB partitioning) is not being compiled.
9 years ago MDEV-6076 Persistent AUTO_INCREMENT for InnoDB
This should be functionally equivalent to WL#6204 in MySQL 8.0.0, with
the notable difference that the file format changes are limited to
repurposing a previously unused data field in B-tree pages.
For persistent InnoDB tables, write the last used AUTO_INCREMENT
value to the root page of the clustered index, in the previously
unused (0) PAGE_MAX_TRX_ID field, now aliased as PAGE_ROOT_AUTO_INC.
Unlike some other previously unused InnoDB data fields, this one was
actually always zero-initialized, at least since MySQL 3.23.49.
The writes to PAGE_ROOT_AUTO_INC are protected by SX or X latch on the
root page. The SX latch will allow concurrent read access to the root
page. (The field PAGE_ROOT_AUTO_INC will only be read on the
first-time call to ha_innobase::open() from the SQL layer. The
PAGE_ROOT_AUTO_INC can only be updated when executing SQL, so
read/write races are not possible.)
During INSERT, the PAGE_ROOT_AUTO_INC is updated by the low-level
function btr_cur_search_to_nth_level(), adding no extra page
access. [Adaptive hash index lookup will be disabled during INSERT.]
If some rare UPDATE modifies an AUTO_INCREMENT column, the
PAGE_ROOT_AUTO_INC will be adjusted in a separate mini-transaction in
ha_innobase::update_row().
When a page is reorganized, we have to preserve the PAGE_ROOT_AUTO_INC
field.
During ALTER TABLE, the initial AUTO_INCREMENT value will be copied
from the table. ALGORITHM=COPY and online log apply in LOCK=NONE will
update PAGE_ROOT_AUTO_INC in real time.
innodb_col_no(): Determine the dict_table_t::cols[] element index
corresponding to a Field of a non-virtual column.
(The MySQL 5.7 implementation of virtual columns breaks the 1:1
relationship between Field::field_index and dict_table_t::cols[].
Virtual columns are omitted from dict_table_t::cols[]. Therefore,
we must translate the field_index of AUTO_INCREMENT columns into
an index of dict_table_t::cols[].)
Upgrade from old data files:
By default, the AUTO_INCREMENT sequence in old data files would appear
to be reset, because PAGE_MAX_TRX_ID or PAGE_ROOT_AUTO_INC would contain
the value 0 in each clustered index page. In new data files,
PAGE_ROOT_AUTO_INC can only be 0 if the table is empty or does not contain
any AUTO_INCREMENT column.
For backward compatibility, we use the old method of
SELECT MAX(auto_increment_column) for initializing the sequence.
btr_read_autoinc(): Read the AUTO_INCREMENT sequence from a new-format
data file.
btr_read_autoinc_with_fallback(): A variant of btr_read_autoinc()
that will resort to reading MAX(auto_increment_column) for data files
that did not use AUTO_INCREMENT yet. It was manually tested that during
the execution of innodb.autoinc_persist the compatibility logic is
not activated (for new files, PAGE_ROOT_AUTO_INC is never 0 in nonempty
clustered index root pages).
initialize_auto_increment(): Replaces
ha_innobase::innobase_initialize_autoinc(). This initializes
the AUTO_INCREMENT metadata. Only called from ha_innobase::open().
ha_innobase::info_low(): Do not try to lazily initialize
dict_table_t::autoinc. It must already have been initialized by
ha_innobase::open() or ha_innobase::create().
Note: The adjustments to class ha_innopart were not tested, because
the source code (native InnoDB partitioning) is not being compiled.
9 years ago  MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
7 years ago MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional
In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.
A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.
MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.
Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.
So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.
We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.
WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.
ut_crc32_byte_by_byte: Remove (unused function).
legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.
buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.
7 years ago MDEV-18644: Support full_crc32 for page_compressed
This is a follow-up task to MDEV-12026, which introduced
innodb_checksum_algorithm=full_crc32 and a simpler page format.
MDEV-12026 did not enable full_crc32 for page_compressed tables,
which we will be doing now.
This is joint work with Thirunarayanan Balathandayuthapani.
For innodb_checksum_algorithm=full_crc32 we change the
page_compressed format as follows:
FIL_PAGE_TYPE: The most significant bit will be set to indicate
page_compressed format. The least significant bits will contain
the compressed page size, rounded up to a multiple of 256 bytes.
The checksum will be stored in the last 4 bytes of the page
(whether it is the full page or a page_compressed page whose
size is determined by FIL_PAGE_TYPE), covering all preceding
bytes of the page. If encryption is used, then the page will
be encrypted between compression and computing the checksum.
For page_compressed, FIL_PAGE_LSN will not be repeated at
the end of the page.
FSP_SPACE_FLAGS (already implemented as part of MDEV-12026):
We will store the innodb_compression_algorithm that may be used
to compress pages. Previously, the choice of algorithm was written
to each compressed data page separately, and one would be unable
to know in advance which compression algorithm(s) are used.
fil_space_t::full_crc32_page_compressed_len(): Determine if the
page_compressed algorithm of the tablespace needs to know the
exact length of the compressed data. If yes, we will reserve and
write an extra byte for this right before the checksum.
buf_page_is_compressed(): Determine if a page uses page_compressed
(in any innodb_checksum_algorithm).
fil_page_decompress(): Pass also fil_space_t::flags so that the
format can be determined.
buf_page_is_zeroes(): Check if a page is full of zero bytes.
buf_page_full_crc32_is_corrupted(): Renamed from
buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32,
we always simply validate the checksum to the page contents,
while the physical page size is explicitly specified by an
unencrypted part of the page header.
buf_page_full_crc32_size(): Determine the size of a full_crc32 page.
buf_dblwr_check_page_lsn(): Make this a debug-only function, because
it involves potentially costly lookups of fil_space_t.
create_table_info_t::check_table_options(),
ha_innobase::check_if_supported_inplace_alter(): Do allow the creation
of SPATIAL INDEX with full_crc32 also when page_compressed is used.
commit_cache_norebuild(): Preserve the compression algorithm when
updating the page_compression_level.
dict_tf_to_fsp_flags(): Set the flags for page compression algorithm.
FIXME: Maybe there should be a table option page_compression_algorithm
and a session variable to back it?
7 years ago MDEV-18644: Support full_crc32 for page_compressed
This is a follow-up task to MDEV-12026, which introduced
innodb_checksum_algorithm=full_crc32 and a simpler page format.
MDEV-12026 did not enable full_crc32 for page_compressed tables,
which we will be doing now.
This is joint work with Thirunarayanan Balathandayuthapani.
For innodb_checksum_algorithm=full_crc32 we change the
page_compressed format as follows:
FIL_PAGE_TYPE: The most significant bit will be set to indicate
page_compressed format. The least significant bits will contain
the compressed page size, rounded up to a multiple of 256 bytes.
The checksum will be stored in the last 4 bytes of the page
(whether it is the full page or a page_compressed page whose
size is determined by FIL_PAGE_TYPE), covering all preceding
bytes of the page. If encryption is used, then the page will
be encrypted between compression and computing the checksum.
For page_compressed, FIL_PAGE_LSN will not be repeated at
the end of the page.
FSP_SPACE_FLAGS (already implemented as part of MDEV-12026):
We will store the innodb_compression_algorithm that may be used
to compress pages. Previously, the choice of algorithm was written
to each compressed data page separately, and one would be unable
to know in advance which compression algorithm(s) are used.
fil_space_t::full_crc32_page_compressed_len(): Determine if the
page_compressed algorithm of the tablespace needs to know the
exact length of the compressed data. If yes, we will reserve and
write an extra byte for this right before the checksum.
buf_page_is_compressed(): Determine if a page uses page_compressed
(in any innodb_checksum_algorithm).
fil_page_decompress(): Pass also fil_space_t::flags so that the
format can be determined.
buf_page_is_zeroes(): Check if a page is full of zero bytes.
buf_page_full_crc32_is_corrupted(): Renamed from
buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32,
we always simply validate the checksum to the page contents,
while the physical page size is explicitly specified by an
unencrypted part of the page header.
buf_page_full_crc32_size(): Determine the size of a full_crc32 page.
buf_dblwr_check_page_lsn(): Make this a debug-only function, because
it involves potentially costly lookups of fil_space_t.
create_table_info_t::check_table_options(),
ha_innobase::check_if_supported_inplace_alter(): Do allow the creation
of SPATIAL INDEX with full_crc32 also when page_compressed is used.
commit_cache_norebuild(): Preserve the compression algorithm when
updating the page_compression_level.
dict_tf_to_fsp_flags(): Set the flags for page compression algorithm.
FIXME: Maybe there should be a table option page_compression_algorithm
and a session variable to back it?
7 years ago MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional
In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.
A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.
MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.
Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.
So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.
We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.
WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.
ut_crc32_byte_by_byte: Remove (unused function).
legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.
buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.
7 years ago MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional
In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.
A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.
MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.
Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.
So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.
We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.
WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.
ut_crc32_byte_by_byte: Remove (unused function).
legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.
buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.
7 years ago MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
7 years ago MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
7 years ago MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional
In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.
A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.
MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.
Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.
So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.
We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.
WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.
ut_crc32_byte_by_byte: Remove (unused function).
legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.
buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.
7 years ago MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional
In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.
A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.
MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.
Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.
So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.
We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.
WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.
ut_crc32_byte_by_byte: Remove (unused function).
legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.
buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.
7 years ago |
|
/*****************************************************************************
Copyright (c) 2005, 2016, Oracle and/or its affiliates. All Rights Reserved.Copyright (c) 2012, Facebook Inc.Copyright (c) 2014, 2019, MariaDB Corporation.
This program is free software; you can redistribute it and/or modify it underthe terms of the GNU General Public License as published by the Free SoftwareFoundation; version 2 of the License.
This program is distributed in the hope that it will be useful, but WITHOUTANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESSFOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along withthis program; if not, write to the Free Software Foundation, Inc.,51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA
*****************************************************************************/
/**************************************************//**
@file page/page0zip.ccCompressed page interface
Created June 2005 by Marko Makela*******************************************************/
#include "page0zip.h"
#include "fsp0types.h"
#include "page0page.h"
#include "buf0checksum.h"
#include "ut0crc32.h"
#include "zlib.h"
#ifndef UNIV_INNOCHECKSUM
/** A BLOB field reference full of zero, for use in assertions and tests.
Initially, BLOB field references are set to zero, indtuple_convert_big_rec(). */const byte field_ref_zero[UNIV_PAGE_SIZE_MAX] = { 0, };
#include "mtr0log.h"
#include "dict0dict.h"
#include "btr0cur.h"
#include "log0recv.h"
#include "row0row.h"
#include "btr0sea.h"
#include "dict0boot.h"
#include "lock0lock.h"
#include "srv0srv.h"
#include "buf0lru.h"
#include "srv0mon.h"
#include <map>
#include <algorithm>
/** Statistics on compression, indexed by page_zip_des_t::ssize - 1 */page_zip_stat_t page_zip_stat[PAGE_ZIP_SSIZE_MAX];/** Statistics on compression, indexed by index->id */page_zip_stat_per_index_t page_zip_stat_per_index;
/** Compression level to be used by zlib. Settable by user. */uint page_zip_level;
/** Whether or not to log compressed page images to avoid possible
compression algorithm changes in zlib. */my_bool page_zip_log_pages;
/* Please refer to ../include/page0zip.ic for a description of the
compressed page format. */
/* The infimum and supremum records are omitted from the compressed page.
On compress, we compare that the records are there, and on uncompress werestore the records. *//** Extra bytes of an infimum record */static const byte infimum_extra[] = { 0x01, /* info_bits=0, n_owned=1 */ 0x00, 0x02 /* heap_no=0, status=2 */ /* ?, ? */ /* next=(first user rec, or supremum) */};/** Data bytes of an infimum record */static const byte infimum_data[] = { 0x69, 0x6e, 0x66, 0x69, 0x6d, 0x75, 0x6d, 0x00 /* "infimum\0" */};/** Extra bytes and data bytes of a supremum record */static const byte supremum_extra_data alignas(4) [] = { /* 0x0?, */ /* info_bits=0, n_owned=1..8 */ 0x00, 0x0b, /* heap_no=1, status=3 */ 0x00, 0x00, /* next=0 */ 0x73, 0x75, 0x70, 0x72, 0x65, 0x6d, 0x75, 0x6d /* "supremum" */};
/** Assert that a block of memory is filled with zero bytes.
Compare at most sizeof(field_ref_zero) bytes.@param b in: memory block@param s in: size of the memory block, in bytes */#define ASSERT_ZERO(b, s) \
ut_ad(!memcmp(b, field_ref_zero, \ std::min<size_t>(s, sizeof field_ref_zero)));/** Assert that a BLOB pointer is filled with zero bytes.
@param b in: BLOB pointer */#define ASSERT_ZERO_BLOB(b) \
ut_ad(!memcmp(b, field_ref_zero, FIELD_REF_SIZE))
/* Enable some extra debugging output. This code can be enabled
independently of any UNIV_ debugging conditions. */#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
# include <stdarg.h>
MY_ATTRIBUTE((format (printf, 1, 2)))/**********************************************************************//**
Report a failure to decompress or compress.@return number of characters printed */staticintpage_zip_fail_func(/*===============*/ const char* fmt, /*!< in: printf(3) format string */ ...) /*!< in: arguments corresponding to fmt */{ int res; va_list ap;
ut_print_timestamp(stderr); fputs(" InnoDB: ", stderr); va_start(ap, fmt); res = vfprintf(stderr, fmt, ap); va_end(ap);
return(res);}/** Wrapper for page_zip_fail_func()
@param fmt_args in: printf(3) format string and arguments */# define page_zip_fail(fmt_args) page_zip_fail_func fmt_args
#else /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
/** Dummy wrapper for page_zip_fail_func()
@param fmt_args ignored: printf(3) format string and arguments */# define page_zip_fail(fmt_args) /* empty */
#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
/**********************************************************************//**
Determine the guaranteed free space on an empty page.@return minimum payload size on the page */ulintpage_zip_empty_size(/*================*/ ulint n_fields, /*!< in: number of columns in the index */ ulint zip_size) /*!< in: compressed page size in bytes */{ ulint size = zip_size /* subtract the page header and the longest
uncompressed data needed for one record */ - (PAGE_DATA + PAGE_ZIP_CLUST_LEAF_SLOT_SIZE + 1/* encoded heap_no==2 in page_zip_write_rec() */ + 1/* end of modification log */ - REC_N_NEW_EXTRA_BYTES/* omitted bytes */) /* subtract the space for page_zip_fields_encode() */ - compressBound(static_cast<uLong>(2 * (n_fields + 1))); return(lint(size) > 0 ? size : 0);}
/** Check whether a tuple is too big for compressed table
@param[in] index dict index object@param[in] entry entry for the index@return true if it's too big, otherwise false */boolpage_zip_is_too_big( const dict_index_t* index, const dtuple_t* entry){ const ulint zip_size = index->table->space->zip_size();
/* Estimate the free space of an empty compressed page.
Subtract one byte for the encoded heap_no in the modification log. */ ulint free_space_zip = page_zip_empty_size( index->n_fields, zip_size); ulint n_uniq = dict_index_get_n_unique_in_tree(index);
ut_ad(dict_table_is_comp(index->table)); ut_ad(zip_size);
if (free_space_zip == 0) { return(true); }
/* Subtract one byte for the encoded heap_no in the
modification log. */ free_space_zip--;
/* There should be enough room for two node pointer
records on an empty non-leaf page. This prevents infinite page splits. */
if (entry->n_fields >= n_uniq && (REC_NODE_PTR_SIZE + rec_get_converted_size_comp_prefix( index, entry->fields, n_uniq, NULL) /* On a compressed page, there is
a two-byte entry in the dense page directory for every record. But there is no record header. */ - (REC_N_NEW_EXTRA_BYTES - 2) > free_space_zip / 2)) { return(true); }
return(false);}
/*************************************************************//**
Gets the number of elements in the dense page directory,including deleted records (the free list).@return number of elements in the dense page directory */UNIV_INLINEulintpage_zip_dir_elems(/*===============*/ const page_zip_des_t* page_zip) /*!< in: compressed page */{ /* Exclude the page infimum and supremum from the record count. */ return ulint(page_dir_get_n_heap(page_zip->data)) - PAGE_HEAP_NO_USER_LOW;}
/*************************************************************//**
Gets the size of the compressed page trailer (the dense page directory),including deleted records (the free list).@return length of dense page directory, in bytes */UNIV_INLINEulintpage_zip_dir_size(/*==============*/ const page_zip_des_t* page_zip) /*!< in: compressed page */{ return(PAGE_ZIP_DIR_SLOT_SIZE * page_zip_dir_elems(page_zip));}
/*************************************************************//**
Gets an offset to the compressed page trailer (the dense page directory),including deleted records (the free list).@return offset of the dense page directory */UNIV_INLINEulintpage_zip_dir_start_offs(/*====================*/ const page_zip_des_t* page_zip, /*!< in: compressed page */ ulint n_dense) /*!< in: directory size */{ ut_ad(n_dense * PAGE_ZIP_DIR_SLOT_SIZE < page_zip_get_size(page_zip));
return(page_zip_get_size(page_zip) - n_dense * PAGE_ZIP_DIR_SLOT_SIZE);}
/*************************************************************//**
Gets a pointer to the compressed page trailer (the dense page directory),including deleted records (the free list).@param[in] page_zip compressed page@param[in] n_dense number of entries in the directory@return pointer to the dense page directory */#define page_zip_dir_start_low(page_zip, n_dense) \
((page_zip)->data + page_zip_dir_start_offs(page_zip, n_dense))/*************************************************************//**
Gets a pointer to the compressed page trailer (the dense page directory),including deleted records (the free list).@param[in] page_zip compressed page@return pointer to the dense page directory */#define page_zip_dir_start(page_zip) \
page_zip_dir_start_low(page_zip, page_zip_dir_elems(page_zip))
/*************************************************************//**
Gets the size of the compressed page trailer (the dense page directory),only including user records (excluding the free list).@return length of dense page directory comprising existing records, in bytes */UNIV_INLINEulintpage_zip_dir_user_size(/*===================*/ const page_zip_des_t* page_zip) /*!< in: compressed page */{ ulint size = PAGE_ZIP_DIR_SLOT_SIZE * ulint(page_get_n_recs(page_zip->data)); ut_ad(size <= page_zip_dir_size(page_zip)); return(size);}
/*************************************************************//**
Find the slot of the given record in the dense page directory.@return dense directory slot, or NULL if record not found */UNIV_INLINEbyte*page_zip_dir_find_low(/*==================*/ byte* slot, /*!< in: start of records */ byte* end, /*!< in: end of records */ ulint offset) /*!< in: offset of user record */{ ut_ad(slot <= end);
for (; slot < end; slot += PAGE_ZIP_DIR_SLOT_SIZE) { if ((mach_read_from_2(slot) & PAGE_ZIP_DIR_SLOT_MASK) == offset) { return(slot); } }
return(NULL);}
/*************************************************************//**
Find the slot of the given non-free record in the dense page directory.@return dense directory slot, or NULL if record not found */UNIV_INLINEbyte*page_zip_dir_find(/*==============*/ page_zip_des_t* page_zip, /*!< in: compressed page */ ulint offset) /*!< in: offset of user record */{ byte* end = page_zip->data + page_zip_get_size(page_zip);
ut_ad(page_zip_simple_validate(page_zip));
return(page_zip_dir_find_low(end - page_zip_dir_user_size(page_zip), end, offset));}
/*************************************************************//**
Find the slot of the given free record in the dense page directory.@return dense directory slot, or NULL if record not found */UNIV_INLINEbyte*page_zip_dir_find_free(/*===================*/ page_zip_des_t* page_zip, /*!< in: compressed page */ ulint offset) /*!< in: offset of user record */{ byte* end = page_zip->data + page_zip_get_size(page_zip);
ut_ad(page_zip_simple_validate(page_zip));
return(page_zip_dir_find_low(end - page_zip_dir_size(page_zip), end - page_zip_dir_user_size(page_zip), offset));}
/*************************************************************//**
Read a given slot in the dense page directory.@return record offset on the uncompressed page, possibly ORed withPAGE_ZIP_DIR_SLOT_DEL or PAGE_ZIP_DIR_SLOT_OWNED */UNIV_INLINEulintpage_zip_dir_get(/*=============*/ const page_zip_des_t* page_zip, /*!< in: compressed page */ ulint slot) /*!< in: slot
(0=first user record) */{ ut_ad(page_zip_simple_validate(page_zip)); ut_ad(slot < page_zip_dir_size(page_zip) / PAGE_ZIP_DIR_SLOT_SIZE); return(mach_read_from_2(page_zip->data + page_zip_get_size(page_zip) - PAGE_ZIP_DIR_SLOT_SIZE * (slot + 1)));}
/** Write a MLOG_ZIP_PAGE_COMPRESS record of compressing an index page.
@param[in,out] block ROW_FORMAT=COMPRESSED index page@param[in] index the index that the block belongs to@param[in,out] mtr mini-transaction */static void page_zip_compress_write_log(buf_block_t* block, dict_index_t* index, mtr_t* mtr){ byte* log_ptr; ulint trailer_size;
ut_ad(!dict_index_is_ibuf(index));
log_ptr = mlog_open(mtr, 11 + 2 + 2);
if (!log_ptr) {
return; }
const page_t* page = block->frame; const page_zip_des_t* page_zip = &block->page.zip; /* Read the number of user records. */ trailer_size = ulint(page_dir_get_n_heap(page_zip->data)) - PAGE_HEAP_NO_USER_LOW; /* Multiply by uncompressed of size stored per record */ if (!page_is_leaf(page)) { trailer_size *= PAGE_ZIP_DIR_SLOT_SIZE + REC_NODE_PTR_SIZE; } else if (dict_index_is_clust(index)) { trailer_size *= PAGE_ZIP_DIR_SLOT_SIZE + DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN; } else { trailer_size *= PAGE_ZIP_DIR_SLOT_SIZE; } /* Add the space occupied by BLOB pointers. */ trailer_size += page_zip->n_blobs * BTR_EXTERN_FIELD_REF_SIZE; ut_a(page_zip->m_end > PAGE_DATA); compile_time_assert(FIL_PAGE_DATA <= PAGE_DATA); ut_a(page_zip->m_end + trailer_size <= page_zip_get_size(page_zip));
log_ptr = mlog_write_initial_log_record_low(MLOG_ZIP_PAGE_COMPRESS, block->page.id.space(), block->page.id.page_no(), log_ptr, mtr); mach_write_to_2(log_ptr, ulint(page_zip->m_end - FIL_PAGE_TYPE)); log_ptr += 2; mach_write_to_2(log_ptr, trailer_size); log_ptr += 2; mlog_close(mtr, log_ptr);
/* Write FIL_PAGE_PREV and FIL_PAGE_NEXT */ mlog_catenate_string(mtr, page_zip->data + FIL_PAGE_PREV, 4); mlog_catenate_string(mtr, page_zip->data + FIL_PAGE_NEXT, 4); /* Write most of the page header, the compressed stream and
the modification log. */ mlog_catenate_string(mtr, page_zip->data + FIL_PAGE_TYPE, ulint(page_zip->m_end - FIL_PAGE_TYPE)); /* Write the uncompressed trailer of the compressed page. */ mlog_catenate_string(mtr, page_zip->data + page_zip_get_size(page_zip) - trailer_size, trailer_size); if (!innodb_log_optimize_ddl) { block->page.init_on_flush = true; }}
/******************************************************//**
Determine how many externally stored columns are containedin existing records with smaller heap_no than rec. */staticulintpage_zip_get_n_prev_extern(/*=======================*/ const page_zip_des_t* page_zip,/*!< in: dense page directory on
compressed page */ const rec_t* rec, /*!< in: compact physical record
on a B-tree leaf page */ const dict_index_t* index) /*!< in: record descriptor */{ const page_t* page = page_align(rec); ulint n_ext = 0; ulint i; ulint left; ulint heap_no; ulint n_recs = page_get_n_recs(page_zip->data);
ut_ad(page_is_leaf(page)); ut_ad(page_is_comp(page)); ut_ad(dict_table_is_comp(index->table)); ut_ad(dict_index_is_clust(index)); ut_ad(!dict_index_is_ibuf(index));
heap_no = rec_get_heap_no_new(rec); ut_ad(heap_no >= PAGE_HEAP_NO_USER_LOW); left = heap_no - PAGE_HEAP_NO_USER_LOW; if (UNIV_UNLIKELY(!left)) { return(0); }
for (i = 0; i < n_recs; i++) { const rec_t* r = page + (page_zip_dir_get(page_zip, i) & PAGE_ZIP_DIR_SLOT_MASK);
if (rec_get_heap_no_new(r) < heap_no) { n_ext += rec_get_n_extern_new(r, index, ULINT_UNDEFINED); if (!--left) { break; } } }
return(n_ext);}
/**********************************************************************//**
Encode the length of a fixed-length column.@return buf + length of encoded val */staticbyte*page_zip_fixed_field_encode(/*========================*/ byte* buf, /*!< in: pointer to buffer where to write */ ulint val) /*!< in: value to write */{ ut_ad(val >= 2);
if (UNIV_LIKELY(val < 126)) { /*
0 = nullable variable field of at most 255 bytes length; 1 = not null variable field of at most 255 bytes length; 126 = nullable variable field with maximum length >255; 127 = not null variable field with maximum length >255 */ *buf++ = (byte) val; } else { *buf++ = (byte) (0x80 | val >> 8); *buf++ = (byte) val; }
return(buf);}
/**********************************************************************//**
Write the index information for the compressed page.@return used size of buf */ulintpage_zip_fields_encode(/*===================*/ ulint n, /*!< in: number of fields
to compress */ const dict_index_t* index, /*!< in: index comprising
at least n fields */ ulint trx_id_pos, /*!< in: position of the trx_id column
in the index, or ULINT_UNDEFINED if this is a non-leaf page */ byte* buf) /*!< out: buffer of (n + 1) * 2 bytes */{ const byte* buf_start = buf; ulint i; ulint col; ulint trx_id_col = 0; /* sum of lengths of preceding non-nullable fixed fields, or 0 */ ulint fixed_sum = 0;
ut_ad(trx_id_pos == ULINT_UNDEFINED || trx_id_pos < n);
for (i = col = 0; i < n; i++) { dict_field_t* field = dict_index_get_nth_field(index, i); ulint val;
if (dict_field_get_col(field)->prtype & DATA_NOT_NULL) { val = 1; /* set the "not nullable" flag */ } else { val = 0; /* nullable field */ }
if (!field->fixed_len) { /* variable-length field */ const dict_col_t* column = dict_field_get_col(field);
if (DATA_BIG_COL(column)) { val |= 0x7e; /* max > 255 bytes */ }
if (fixed_sum) { /* write out the length of any
preceding non-nullable fields */ buf = page_zip_fixed_field_encode( buf, fixed_sum << 1 | 1); fixed_sum = 0; col++; }
*buf++ = (byte) val; col++; } else if (val) { /* fixed-length non-nullable field */
if (fixed_sum && UNIV_UNLIKELY (fixed_sum + field->fixed_len > DICT_MAX_FIXED_COL_LEN)) { /* Write out the length of the
preceding non-nullable fields, to avoid exceeding the maximum length of a fixed-length column. */ buf = page_zip_fixed_field_encode( buf, fixed_sum << 1 | 1); fixed_sum = 0; col++; }
if (i && UNIV_UNLIKELY(i == trx_id_pos)) { if (fixed_sum) { /* Write out the length of any
preceding non-nullable fields, and start a new trx_id column. */ buf = page_zip_fixed_field_encode( buf, fixed_sum << 1 | 1); col++; }
trx_id_col = col; fixed_sum = field->fixed_len; } else { /* add to the sum */ fixed_sum += field->fixed_len; } } else { /* fixed-length nullable field */
if (fixed_sum) { /* write out the length of any
preceding non-nullable fields */ buf = page_zip_fixed_field_encode( buf, fixed_sum << 1 | 1); fixed_sum = 0; col++; }
buf = page_zip_fixed_field_encode( buf, ulint(field->fixed_len) << 1); col++; } }
if (fixed_sum) { /* Write out the lengths of last fixed-length columns. */ buf = page_zip_fixed_field_encode(buf, fixed_sum << 1 | 1); }
if (trx_id_pos != ULINT_UNDEFINED) { /* Write out the position of the trx_id column */ i = trx_id_col; } else { /* Write out the number of nullable fields */ i = index->n_nullable; }
if (i < 128) { *buf++ = (byte) i; } else { *buf++ = (byte) (0x80 | i >> 8); *buf++ = (byte) i; }
ut_ad((ulint) (buf - buf_start) <= (n + 2) * 2); return((ulint) (buf - buf_start));}
/**********************************************************************//**
Populate the dense page directory from the sparse directory. */staticvoidpage_zip_dir_encode(/*================*/ const page_t* page, /*!< in: compact page */ byte* buf, /*!< in: pointer to dense page directory[-1];
out: dense directory on compressed page */ const rec_t** recs) /*!< in: pointer to an array of 0, or NULL;
out: dense page directory sorted by ascending address (and heap_no) */{ const byte* rec; ulint status; ulint min_mark; ulint heap_no; ulint i; ulint n_heap; ulint offs;
min_mark = 0;
if (page_is_leaf(page)) { status = REC_STATUS_ORDINARY; } else { status = REC_STATUS_NODE_PTR; if (UNIV_UNLIKELY(!page_has_prev(page))) { min_mark = REC_INFO_MIN_REC_FLAG; } }
n_heap = page_dir_get_n_heap(page);
/* Traverse the list of stored records in the collation order,
starting from the first user record. */
rec = page + PAGE_NEW_INFIMUM;
i = 0;
for (;;) { ulint info_bits; offs = rec_get_next_offs(rec, TRUE); if (UNIV_UNLIKELY(offs == PAGE_NEW_SUPREMUM)) { break; } rec = page + offs; heap_no = rec_get_heap_no_new(rec); ut_a(heap_no >= PAGE_HEAP_NO_USER_LOW); ut_a(heap_no < n_heap); ut_a(offs < srv_page_size - PAGE_DIR); ut_a(offs >= PAGE_ZIP_START); compile_time_assert(!(PAGE_ZIP_DIR_SLOT_MASK & (PAGE_ZIP_DIR_SLOT_MASK + 1))); compile_time_assert(PAGE_ZIP_DIR_SLOT_MASK >= UNIV_ZIP_SIZE_MAX - 1);
if (UNIV_UNLIKELY(rec_get_n_owned_new(rec) != 0)) { offs |= PAGE_ZIP_DIR_SLOT_OWNED; }
info_bits = rec_get_info_bits(rec, TRUE); if (info_bits & REC_INFO_DELETED_FLAG) { info_bits &= ~REC_INFO_DELETED_FLAG; offs |= PAGE_ZIP_DIR_SLOT_DEL; } ut_a(info_bits == min_mark); /* Only the smallest user record can have
REC_INFO_MIN_REC_FLAG set. */ min_mark = 0;
mach_write_to_2(buf - PAGE_ZIP_DIR_SLOT_SIZE * ++i, offs);
if (UNIV_LIKELY_NULL(recs)) { /* Ensure that each heap_no occurs at most once. */ ut_a(!recs[heap_no - PAGE_HEAP_NO_USER_LOW]); /* exclude infimum and supremum */ recs[heap_no - PAGE_HEAP_NO_USER_LOW] = rec; }
ut_a(ulint(rec_get_status(rec)) == status); }
offs = page_header_get_field(page, PAGE_FREE);
/* Traverse the free list (of deleted records). */ while (offs) { ut_ad(!(offs & ~PAGE_ZIP_DIR_SLOT_MASK)); rec = page + offs;
heap_no = rec_get_heap_no_new(rec); ut_a(heap_no >= PAGE_HEAP_NO_USER_LOW); ut_a(heap_no < n_heap);
ut_a(!rec[-REC_N_NEW_EXTRA_BYTES]); /* info_bits and n_owned */ ut_a(ulint(rec_get_status(rec)) == status);
mach_write_to_2(buf - PAGE_ZIP_DIR_SLOT_SIZE * ++i, offs);
if (UNIV_LIKELY_NULL(recs)) { /* Ensure that each heap_no occurs at most once. */ ut_a(!recs[heap_no - PAGE_HEAP_NO_USER_LOW]); /* exclude infimum and supremum */ recs[heap_no - PAGE_HEAP_NO_USER_LOW] = rec; }
offs = rec_get_next_offs(rec, TRUE); }
/* Ensure that each heap no occurs at least once. */ ut_a(i + PAGE_HEAP_NO_USER_LOW == n_heap);}
extern "C" {
/**********************************************************************//**
Allocate memory for zlib. */staticvoid*page_zip_zalloc(/*============*/ void* opaque, /*!< in/out: memory heap */ uInt items, /*!< in: number of items to allocate */ uInt size) /*!< in: size of an item in bytes */{ return(mem_heap_zalloc(static_cast<mem_heap_t*>(opaque), items * size));}
/**********************************************************************//**
Deallocate memory for zlib. */staticvoidpage_zip_free(/*==========*/ void* opaque MY_ATTRIBUTE((unused)), /*!< in: memory heap */ void* address MY_ATTRIBUTE((unused)))/*!< in: object to free */{}
} /* extern "C" */
/**********************************************************************//**
Configure the zlib allocator to use the given memory heap. */voidpage_zip_set_alloc(/*===============*/ void* stream, /*!< in/out: zlib stream */ mem_heap_t* heap) /*!< in: memory heap to use */{ z_stream* strm = static_cast<z_stream*>(stream);
strm->zalloc = page_zip_zalloc; strm->zfree = page_zip_free; strm->opaque = heap;}
#if 0 || defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
/** Symbol for enabling compression and decompression diagnostics */# define PAGE_ZIP_COMPRESS_DBG
#endif
#ifdef PAGE_ZIP_COMPRESS_DBG
/** Set this variable in a debugger to enable
excessive logging in page_zip_compress(). */static bool page_zip_compress_dbg;/** Set this variable in a debugger to enable
binary logging of the data passed to deflate().When this variable is nonzero, it will actas a log file name generator. */static unsigned page_zip_compress_log;
/**********************************************************************//**
Wrapper for deflate(). Log the operation if page_zip_compress_dbg is set.@return deflate() status: Z_OK, Z_BUF_ERROR, ... */staticintpage_zip_compress_deflate(/*======================*/ FILE* logfile,/*!< in: log file, or NULL */ z_streamp strm, /*!< in/out: compressed stream for deflate() */ int flush) /*!< in: deflate() flushing method */{ int status; if (UNIV_UNLIKELY(page_zip_compress_dbg)) { ut_print_buf(stderr, strm->next_in, strm->avail_in); } if (UNIV_LIKELY_NULL(logfile)) { if (fwrite(strm->next_in, 1, strm->avail_in, logfile) != strm->avail_in) { perror("fwrite"); } } status = deflate(strm, flush); if (UNIV_UNLIKELY(page_zip_compress_dbg)) { fprintf(stderr, " -> %d\n", status); } return(status);}
/* Redefine deflate(). */# undef deflate
/** Debug wrapper for the zlib compression routine deflate().
Log the operation if page_zip_compress_dbg is set.@param strm in/out: compressed stream@param flush in: flushing method@return deflate() status: Z_OK, Z_BUF_ERROR, ... */# define deflate(strm, flush) page_zip_compress_deflate(logfile, strm, flush)
/** Declaration of the logfile parameter */# define FILE_LOGFILE FILE* logfile,
/** The logfile parameter */# define LOGFILE logfile,
#else /* PAGE_ZIP_COMPRESS_DBG */
/** Empty declaration of the logfile parameter */# define FILE_LOGFILE
/** Missing logfile parameter */# define LOGFILE
#endif /* PAGE_ZIP_COMPRESS_DBG */
/**********************************************************************//**
Compress the records of a node pointer page.@return Z_OK, or a zlib error code */staticintpage_zip_compress_node_ptrs(/*========================*/ FILE_LOGFILE z_stream* c_stream, /*!< in/out: compressed page stream */ const rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense, /*!< in: size of recs[] */ dict_index_t* index, /*!< in: the index of the page */ byte* storage, /*!< in: end of dense page directory */ mem_heap_t* heap) /*!< in: temporary memory heap */{ int err = Z_OK; offset_t* offsets = NULL;
do { const rec_t* rec = *recs++;
offsets = rec_get_offsets(rec, index, offsets, false, ULINT_UNDEFINED, &heap); /* Only leaf nodes may contain externally stored columns. */ ut_ad(!rec_offs_any_extern(offsets));
UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
/* Compress the extra bytes. */ c_stream->avail_in = static_cast<uInt>( rec - REC_N_NEW_EXTRA_BYTES - c_stream->next_in);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) { break; } } ut_ad(!c_stream->avail_in);
/* Compress the data bytes, except node_ptr. */ c_stream->next_in = (byte*) rec; c_stream->avail_in = static_cast<uInt>( rec_offs_data_size(offsets) - REC_NODE_PTR_SIZE);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) { break; } }
ut_ad(!c_stream->avail_in);
memcpy(storage - REC_NODE_PTR_SIZE * (rec_get_heap_no_new(rec) - 1), c_stream->next_in, REC_NODE_PTR_SIZE); c_stream->next_in += REC_NODE_PTR_SIZE; } while (--n_dense);
return(err);}
/**********************************************************************//**
Compress the records of a leaf node of a secondary index.@return Z_OK, or a zlib error code */staticintpage_zip_compress_sec(/*==================*/ FILE_LOGFILE z_stream* c_stream, /*!< in/out: compressed page stream */ const rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense) /*!< in: size of recs[] */{ int err = Z_OK;
ut_ad(n_dense > 0);
do { const rec_t* rec = *recs++;
/* Compress everything up to this record. */ c_stream->avail_in = static_cast<uInt>( rec - REC_N_NEW_EXTRA_BYTES - c_stream->next_in);
if (UNIV_LIKELY(c_stream->avail_in != 0)) { UNIV_MEM_ASSERT_RW(c_stream->next_in, c_stream->avail_in); err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) { break; } }
ut_ad(!c_stream->avail_in); ut_ad(c_stream->next_in == rec - REC_N_NEW_EXTRA_BYTES);
/* Skip the REC_N_NEW_EXTRA_BYTES. */
c_stream->next_in = (byte*) rec; } while (--n_dense);
return(err);}
/**********************************************************************//**
Compress a record of a leaf node of a clustered index that containsexternally stored columns.@return Z_OK, or a zlib error code */staticintpage_zip_compress_clust_ext(/*========================*/ FILE_LOGFILE z_stream* c_stream, /*!< in/out: compressed page stream */ const rec_t* rec, /*!< in: record */ const offset_t* offsets, /*!< in: rec_get_offsets(rec) */ ulint trx_id_col, /*!< in: position of of DB_TRX_ID */ byte* deleted, /*!< in: dense directory entry pointing
to the head of the free list */ byte* storage, /*!< in: end of dense page directory */ byte** externs, /*!< in/out: pointer to the next
available BLOB pointer */ ulint* n_blobs) /*!< in/out: number of
externally stored columns */{ int err; ulint i;
UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
for (i = 0; i < rec_offs_n_fields(offsets); i++) { ulint len; const byte* src;
if (UNIV_UNLIKELY(i == trx_id_col)) { ut_ad(!rec_offs_nth_extern(offsets, i)); /* Store trx_id and roll_ptr
in uncompressed form. */ src = rec_get_nth_field(rec, offsets, i, &len); ut_ad(src + DATA_TRX_ID_LEN == rec_get_nth_field(rec, offsets, i + 1, &len)); ut_ad(len == DATA_ROLL_PTR_LEN);
/* Compress any preceding bytes. */ c_stream->avail_in = static_cast<uInt>( src - c_stream->next_in);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) {
return(err); } }
ut_ad(!c_stream->avail_in); ut_ad(c_stream->next_in == src);
memcpy(storage - (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) * (rec_get_heap_no_new(rec) - 1), c_stream->next_in, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
c_stream->next_in += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN;
/* Skip also roll_ptr */ i++; } else if (rec_offs_nth_extern(offsets, i)) { src = rec_get_nth_field(rec, offsets, i, &len); ut_ad(len >= BTR_EXTERN_FIELD_REF_SIZE); src += len - BTR_EXTERN_FIELD_REF_SIZE;
c_stream->avail_in = static_cast<uInt>( src - c_stream->next_in); if (UNIV_LIKELY(c_stream->avail_in != 0)) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) {
return(err); } }
ut_ad(!c_stream->avail_in); ut_ad(c_stream->next_in == src);
/* Reserve space for the data at
the end of the space reserved for the compressed data and the page modification log. */
if (UNIV_UNLIKELY (c_stream->avail_out <= BTR_EXTERN_FIELD_REF_SIZE)) { /* out of space */ return(Z_BUF_ERROR); }
ut_ad(*externs == c_stream->next_out + c_stream->avail_out + 1/* end of modif. log */);
c_stream->next_in += BTR_EXTERN_FIELD_REF_SIZE;
/* Skip deleted records. */ if (UNIV_LIKELY_NULL (page_zip_dir_find_low( storage, deleted, page_offset(rec)))) { continue; }
(*n_blobs)++; c_stream->avail_out -= BTR_EXTERN_FIELD_REF_SIZE; *externs -= BTR_EXTERN_FIELD_REF_SIZE;
/* Copy the BLOB pointer */ memcpy(*externs, c_stream->next_in - BTR_EXTERN_FIELD_REF_SIZE, BTR_EXTERN_FIELD_REF_SIZE); } }
return(Z_OK);}
/**********************************************************************//**
Compress the records of a leaf node of a clustered index.@return Z_OK, or a zlib error code */staticintpage_zip_compress_clust(/*====================*/ FILE_LOGFILE z_stream* c_stream, /*!< in/out: compressed page stream */ const rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense, /*!< in: size of recs[] */ dict_index_t* index, /*!< in: the index of the page */ ulint* n_blobs, /*!< in: 0; out: number of
externally stored columns */ ulint trx_id_col, /*!< index of the trx_id column */ byte* deleted, /*!< in: dense directory entry pointing
to the head of the free list */ byte* storage, /*!< in: end of dense page directory */ mem_heap_t* heap) /*!< in: temporary memory heap */{ int err = Z_OK; offset_t* offsets = NULL; /* BTR_EXTERN_FIELD_REF storage */ byte* externs = storage - n_dense * (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
ut_ad(*n_blobs == 0);
do { const rec_t* rec = *recs++;
offsets = rec_get_offsets(rec, index, offsets, true, ULINT_UNDEFINED, &heap); ut_ad(rec_offs_n_fields(offsets) == dict_index_get_n_fields(index)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
/* Compress the extra bytes. */ c_stream->avail_in = static_cast<uInt>( rec - REC_N_NEW_EXTRA_BYTES - c_stream->next_in);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) {
goto func_exit; } } ut_ad(!c_stream->avail_in); ut_ad(c_stream->next_in == rec - REC_N_NEW_EXTRA_BYTES);
/* Compress the data bytes. */
c_stream->next_in = (byte*) rec;
/* Check if there are any externally stored columns.
For each externally stored column, store the BTR_EXTERN_FIELD_REF separately. */ if (rec_offs_any_extern(offsets)) { ut_ad(dict_index_is_clust(index));
err = page_zip_compress_clust_ext( LOGFILE c_stream, rec, offsets, trx_id_col, deleted, storage, &externs, n_blobs);
if (UNIV_UNLIKELY(err != Z_OK)) {
goto func_exit; } } else { ulint len; const byte* src;
/* Store trx_id and roll_ptr in uncompressed form. */ src = rec_get_nth_field(rec, offsets, trx_id_col, &len); ut_ad(src + DATA_TRX_ID_LEN == rec_get_nth_field(rec, offsets, trx_id_col + 1, &len)); ut_ad(len == DATA_ROLL_PTR_LEN); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
/* Compress any preceding bytes. */ c_stream->avail_in = static_cast<uInt>( src - c_stream->next_in);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) {
return(err); } }
ut_ad(!c_stream->avail_in); ut_ad(c_stream->next_in == src);
memcpy(storage - (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) * (rec_get_heap_no_new(rec) - 1), c_stream->next_in, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
c_stream->next_in += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN;
/* Skip also roll_ptr */ ut_ad(trx_id_col + 1 < rec_offs_n_fields(offsets)); }
/* Compress the last bytes of the record. */ c_stream->avail_in = static_cast<uInt>( rec + rec_offs_data_size(offsets) - c_stream->next_in);
if (c_stream->avail_in) { err = deflate(c_stream, Z_NO_FLUSH); if (UNIV_UNLIKELY(err != Z_OK)) {
goto func_exit; } } ut_ad(!c_stream->avail_in); } while (--n_dense);
func_exit: return(err);}
/** Attempt to compress a ROW_FORMAT=COMPRESSED page.
@retval true on success@retval false on failure; block->page.zip will be left intact. */boolpage_zip_compress( buf_block_t* block, /*!< in/out: buffer block */ dict_index_t* index, /*!< in: index of the B-tree node */ ulint level, /*!< in: commpression level */ mtr_t* mtr) /*!< in/out: mini-transaction */{ z_stream c_stream; int err; byte* fields; /*!< index field information */ byte* buf; /*!< compressed payload of the
page */ byte* buf_end; /* end of buf */ ulint n_dense; ulint slot_size; /* amount of uncompressed bytes
per record */ const rec_t** recs; /*!< dense page directory,
sorted by address */ mem_heap_t* heap; ulint trx_id_col = ULINT_UNDEFINED; ulint n_blobs = 0; byte* storage; /* storage of uncompressed
columns */ const ulonglong ns = my_interval_timer();#ifdef PAGE_ZIP_COMPRESS_DBG
FILE* logfile = NULL;#endif
/* A local copy of srv_cmp_per_index_enabled to avoid reading that
variable multiple times in this function since it can be changed at anytime. */ my_bool cmp_per_index_enabled; cmp_per_index_enabled = srv_cmp_per_index_enabled;
page_t* page = block->frame; page_zip_des_t* page_zip = &block->page.zip;
ut_a(page_is_comp(page)); ut_a(fil_page_index_page_check(page)); ut_ad(page_simple_validate_new((page_t*) page)); ut_ad(page_zip_simple_validate(page_zip)); ut_ad(dict_table_is_comp(index->table)); ut_ad(!dict_index_is_ibuf(index));
UNIV_MEM_ASSERT_RW(page, srv_page_size);
/* Check the data that will be omitted. */ ut_a(!memcmp(page + (PAGE_NEW_INFIMUM - REC_N_NEW_EXTRA_BYTES), infimum_extra, sizeof infimum_extra)); ut_a(!memcmp(page + PAGE_NEW_INFIMUM, infimum_data, sizeof infimum_data)); ut_a(page[PAGE_NEW_SUPREMUM - REC_N_NEW_EXTRA_BYTES] /* info_bits == 0, n_owned <= max */ <= PAGE_DIR_SLOT_MAX_N_OWNED); ut_a(!memcmp(page + (PAGE_NEW_SUPREMUM - REC_N_NEW_EXTRA_BYTES + 1), supremum_extra_data, sizeof supremum_extra_data));
if (page_is_empty(page)) { ut_a(rec_get_next_offs(page + PAGE_NEW_INFIMUM, TRUE) == PAGE_NEW_SUPREMUM); }
const ulint n_fields = page_is_leaf(page) ? dict_index_get_n_fields(index) : dict_index_get_n_unique_in_tree_nonleaf(index); index_id_t ind_id = index->id;
/* The dense directory excludes the infimum and supremum records. */ n_dense = ulint(page_dir_get_n_heap(page)) - PAGE_HEAP_NO_USER_LOW;#ifdef PAGE_ZIP_COMPRESS_DBG
if (UNIV_UNLIKELY(page_zip_compress_dbg)) { ib::info() << "compress " << static_cast<void*>(page_zip) << " " << static_cast<const void*>(page) << " " << page_is_leaf(page) << " " << n_fields << " " << n_dense; }
if (UNIV_UNLIKELY(page_zip_compress_log)) { /* Create a log file for every compression attempt. */ char logfilename[9]; snprintf(logfilename, sizeof logfilename, "%08x", page_zip_compress_log++); logfile = fopen(logfilename, "wb");
if (logfile) { /* Write the uncompressed page to the log. */ if (fwrite(page, 1, srv_page_size, logfile) != srv_page_size) { perror("fwrite"); } /* Record the compressed size as zero.
This will be overwritten at successful exit. */ putc(0, logfile); putc(0, logfile); putc(0, logfile); putc(0, logfile); } }#endif /* PAGE_ZIP_COMPRESS_DBG */
page_zip_stat[page_zip->ssize - 1].compressed++; if (cmp_per_index_enabled) { mutex_enter(&page_zip_stat_per_index_mutex); page_zip_stat_per_index[ind_id].compressed++; mutex_exit(&page_zip_stat_per_index_mutex); }
if (UNIV_UNLIKELY(n_dense * PAGE_ZIP_DIR_SLOT_SIZE >= page_zip_get_size(page_zip))) {
goto err_exit; }
MONITOR_INC(MONITOR_PAGE_COMPRESS);
/* Simulate a compression failure with a probability determined by
innodb_simulate_comp_failures, only if the page has 2 or more records. */
if (srv_simulate_comp_failures && !dict_index_is_ibuf(index) && page_get_n_recs(page) >= 2 && ((ulint)(rand() % 100) < srv_simulate_comp_failures) && strcmp(index->table->name.m_name, "IBUF_DUMMY")) {
#ifdef UNIV_DEBUG
ib::error() << "Simulating a compression failure" << " for table " << index->table->name << " index " << index->name() << " page " << block->page.id.page_no() << "(" << (page_is_leaf(page) ? "leaf" : "non-leaf") << ")";
#endif
goto err_exit; }
heap = mem_heap_create(page_zip_get_size(page_zip) + n_fields * (2 + sizeof(ulint)) + REC_OFFS_HEADER_SIZE + n_dense * ((sizeof *recs) - PAGE_ZIP_DIR_SLOT_SIZE) + srv_page_size * 4 + (512 << MAX_MEM_LEVEL));
recs = static_cast<const rec_t**>( mem_heap_zalloc(heap, n_dense * sizeof *recs));
fields = static_cast<byte*>(mem_heap_alloc(heap, (n_fields + 1) * 2));
buf = static_cast<byte*>( mem_heap_alloc(heap, page_zip_get_size(page_zip) - PAGE_DATA));
buf_end = buf + page_zip_get_size(page_zip) - PAGE_DATA;
/* Compress the data payload. */ page_zip_set_alloc(&c_stream, heap);
err = deflateInit2(&c_stream, static_cast<int>(level), Z_DEFLATED, srv_page_size_shift, MAX_MEM_LEVEL, Z_DEFAULT_STRATEGY); ut_a(err == Z_OK);
c_stream.next_out = buf;
/* Subtract the space reserved for uncompressed data. */ /* Page header and the end marker of the modification log */ c_stream.avail_out = static_cast<uInt>(buf_end - buf - 1);
/* Dense page directory and uncompressed columns, if any */ if (page_is_leaf(page)) { if (dict_index_is_clust(index)) { trx_id_col = index->db_trx_id();
slot_size = PAGE_ZIP_DIR_SLOT_SIZE + DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN;
} else { /* Signal the absence of trx_id
in page_zip_fields_encode() */ trx_id_col = 0; slot_size = PAGE_ZIP_DIR_SLOT_SIZE; } } else { slot_size = PAGE_ZIP_DIR_SLOT_SIZE + REC_NODE_PTR_SIZE; trx_id_col = ULINT_UNDEFINED; }
if (UNIV_UNLIKELY(c_stream.avail_out <= n_dense * slot_size + 6/* sizeof(zlib header and footer) */)) { goto zlib_error; }
c_stream.avail_out -= uInt(n_dense * slot_size); c_stream.avail_in = uInt(page_zip_fields_encode(n_fields, index, trx_id_col, fields)); c_stream.next_in = fields;
if (UNIV_LIKELY(!trx_id_col)) { trx_id_col = ULINT_UNDEFINED; }
UNIV_MEM_ASSERT_RW(c_stream.next_in, c_stream.avail_in); err = deflate(&c_stream, Z_FULL_FLUSH); if (err != Z_OK) { goto zlib_error; }
ut_ad(!c_stream.avail_in);
page_zip_dir_encode(page, buf_end, recs);
c_stream.next_in = (byte*) page + PAGE_ZIP_START;
storage = buf_end - n_dense * PAGE_ZIP_DIR_SLOT_SIZE;
/* Compress the records in heap_no order. */ if (UNIV_UNLIKELY(!n_dense)) { } else if (!page_is_leaf(page)) { /* This is a node pointer page. */ err = page_zip_compress_node_ptrs(LOGFILE &c_stream, recs, n_dense, index, storage, heap); if (UNIV_UNLIKELY(err != Z_OK)) { goto zlib_error; } } else if (UNIV_LIKELY(trx_id_col == ULINT_UNDEFINED)) { /* This is a leaf page in a secondary index. */ err = page_zip_compress_sec(LOGFILE &c_stream, recs, n_dense); if (UNIV_UNLIKELY(err != Z_OK)) { goto zlib_error; } } else { /* This is a leaf page in a clustered index. */ err = page_zip_compress_clust(LOGFILE &c_stream, recs, n_dense, index, &n_blobs, trx_id_col, buf_end - PAGE_ZIP_DIR_SLOT_SIZE * page_get_n_recs(page), storage, heap); if (UNIV_UNLIKELY(err != Z_OK)) { goto zlib_error; } }
/* Finish the compression. */ ut_ad(!c_stream.avail_in); /* Compress any trailing garbage, in case the last record was
allocated from an originally longer space on the free list, or the data of the last record from page_zip_compress_sec(). */ c_stream.avail_in = static_cast<uInt>( page_header_get_field(page, PAGE_HEAP_TOP) - (c_stream.next_in - page)); ut_a(c_stream.avail_in <= srv_page_size - PAGE_ZIP_START - PAGE_DIR);
UNIV_MEM_ASSERT_RW(c_stream.next_in, c_stream.avail_in); err = deflate(&c_stream, Z_FINISH);
if (UNIV_UNLIKELY(err != Z_STREAM_END)) {zlib_error: deflateEnd(&c_stream); mem_heap_free(heap);err_exit:#ifdef PAGE_ZIP_COMPRESS_DBG
if (logfile) { fclose(logfile); }#endif /* PAGE_ZIP_COMPRESS_DBG */
if (page_is_leaf(page)) { dict_index_zip_failure(index); }
const uint64_t time_diff = (my_interval_timer() - ns) / 1000; page_zip_stat[page_zip->ssize - 1].compressed_usec += time_diff; if (cmp_per_index_enabled) { mutex_enter(&page_zip_stat_per_index_mutex); page_zip_stat_per_index[ind_id].compressed_usec += time_diff; mutex_exit(&page_zip_stat_per_index_mutex); } return false; }
err = deflateEnd(&c_stream); ut_a(err == Z_OK);
ut_ad(buf + c_stream.total_out == c_stream.next_out); ut_ad((ulint) (storage - c_stream.next_out) >= c_stream.avail_out);
/* Valgrind believes that zlib does not initialize some bits
in the last 7 or 8 bytes of the stream. Make Valgrind happy. */ UNIV_MEM_VALID(buf, c_stream.total_out);
/* Zero out the area reserved for the modification log.
Space for the end marker of the modification log is not included in avail_out. */ memset(c_stream.next_out, 0, c_stream.avail_out + 1/* end marker */);
#ifdef UNIV_DEBUG
page_zip->m_start =#endif /* UNIV_DEBUG */
page_zip->m_end = unsigned(PAGE_DATA + c_stream.total_out); page_zip->m_nonempty = FALSE; page_zip->n_blobs = unsigned(n_blobs); /* Copy those header fields that will not be written
in buf_flush_init_for_writing() */ memcpy_aligned<8>(page_zip->data + FIL_PAGE_PREV, page + FIL_PAGE_PREV, FIL_PAGE_LSN - FIL_PAGE_PREV); memcpy_aligned<2>(page_zip->data + FIL_PAGE_TYPE, page + FIL_PAGE_TYPE, 2); memcpy_aligned<2>(page_zip->data + FIL_PAGE_DATA, page + FIL_PAGE_DATA, PAGE_DATA - FIL_PAGE_DATA); /* Copy the rest of the compressed page */ memcpy_aligned<2>(page_zip->data + PAGE_DATA, buf, page_zip_get_size(page_zip) - PAGE_DATA); mem_heap_free(heap);#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, index));#endif /* UNIV_ZIP_DEBUG */
page_zip_compress_write_log(block, index, mtr);
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
#ifdef PAGE_ZIP_COMPRESS_DBG
if (logfile) { /* Record the compressed size of the block. */ byte sz[4]; mach_write_to_4(sz, c_stream.total_out); fseek(logfile, srv_page_size, SEEK_SET); if (fwrite(sz, 1, sizeof sz, logfile) != sizeof sz) { perror("fwrite"); } fclose(logfile); }#endif /* PAGE_ZIP_COMPRESS_DBG */
const uint64_t time_diff = (my_interval_timer() - ns) / 1000; page_zip_stat[page_zip->ssize - 1].compressed_ok++; page_zip_stat[page_zip->ssize - 1].compressed_usec += time_diff; if (cmp_per_index_enabled) { mutex_enter(&page_zip_stat_per_index_mutex); page_zip_stat_per_index[ind_id].compressed_ok++; page_zip_stat_per_index[ind_id].compressed_usec += time_diff; mutex_exit(&page_zip_stat_per_index_mutex); }
if (page_is_leaf(page)) { dict_index_zip_success(index); }
return true;}
/**********************************************************************//**
Deallocate the index information initialized by page_zip_fields_decode(). */staticvoidpage_zip_fields_free(/*=================*/ dict_index_t* index) /*!< in: dummy index to be freed */{ if (index) { dict_table_t* table = index->table; mutex_free(&index->zip_pad.mutex); mem_heap_free(index->heap);
dict_mem_table_free(table); }}
/**********************************************************************//**
Read the index information for the compressed page.@return own: dummy index describing the page, or NULL on error */staticdict_index_t*page_zip_fields_decode(/*===================*/ const byte* buf, /*!< in: index information */ const byte* end, /*!< in: end of buf */ ulint* trx_id_col,/*!< in: NULL for non-leaf pages;
for leaf pages, pointer to where to store the position of the trx_id column */ bool is_spatial)/*< in: is spatial index or not */{ const byte* b; ulint n; ulint i; ulint val; dict_table_t* table; dict_index_t* index;
/* Determine the number of fields. */ for (b = buf, n = 0; b < end; n++) { if (*b++ & 0x80) { b++; /* skip the second byte */ } }
n--; /* n_nullable or trx_id */
if (UNIV_UNLIKELY(n > REC_MAX_N_FIELDS)) {
page_zip_fail(("page_zip_fields_decode: n = %lu\n", (ulong) n)); return(NULL); }
if (UNIV_UNLIKELY(b > end)) {
page_zip_fail(("page_zip_fields_decode: %p > %p\n", (const void*) b, (const void*) end)); return(NULL); }
table = dict_mem_table_create("ZIP_DUMMY", NULL, n, 0, DICT_TF_COMPACT, 0); index = dict_mem_index_create(table, "ZIP_DUMMY", 0, n); index->n_uniq = unsigned(n); /* avoid ut_ad(index->cached) in dict_index_get_n_unique_in_tree */ index->cached = TRUE;
/* Initialize the fields. */ for (b = buf, i = 0; i < n; i++) { ulint mtype; ulint len;
val = *b++;
if (UNIV_UNLIKELY(val & 0x80)) { /* fixed length > 62 bytes */ val = (val & 0x7f) << 8 | *b++; len = val >> 1; mtype = DATA_FIXBINARY; } else if (UNIV_UNLIKELY(val >= 126)) { /* variable length with max > 255 bytes */ len = 0x7fff; mtype = DATA_BINARY; } else if (val <= 1) { /* variable length with max <= 255 bytes */ len = 0; mtype = DATA_BINARY; } else { /* fixed length < 62 bytes */ len = val >> 1; mtype = DATA_FIXBINARY; }
dict_mem_table_add_col(table, NULL, NULL, mtype, val & 1 ? DATA_NOT_NULL : 0, len); dict_index_add_col(index, table, dict_table_get_nth_col(table, i), 0); }
val = *b++; if (UNIV_UNLIKELY(val & 0x80)) { val = (val & 0x7f) << 8 | *b++; }
/* Decode the position of the trx_id column. */ if (trx_id_col) { if (!val) { val = ULINT_UNDEFINED; } else if (UNIV_UNLIKELY(val >= n)) { page_zip_fields_free(index); index = NULL; } else { index->type = DICT_CLUSTERED; }
*trx_id_col = val; } else { /* Decode the number of nullable fields. */ if (UNIV_UNLIKELY(index->n_nullable > val)) { page_zip_fields_free(index); index = NULL; } else { index->n_nullable = unsigned(val); } }
/* ROW_FORMAT=COMPRESSED does not support instant ADD COLUMN */ index->n_core_fields = index->n_fields; index->n_core_null_bytes = UT_BITS_IN_BYTES(unsigned(index->n_nullable));
ut_ad(b == end);
if (is_spatial) { index->type |= DICT_SPATIAL; }
return(index);}
/**********************************************************************//**
Populate the sparse page directory from the dense directory.@return TRUE on success, FALSE on failure */static MY_ATTRIBUTE((nonnull, warn_unused_result))iboolpage_zip_dir_decode(/*================*/ const page_zip_des_t* page_zip,/*!< in: dense page directory on
compressed page */ page_t* page, /*!< in: compact page with valid header;
out: trailer and sparse page directory filled in */ rec_t** recs, /*!< out: dense page directory sorted by
ascending address (and heap_no) */ ulint n_dense)/*!< in: number of user records, and
size of recs[] */{ ulint i; ulint n_recs; byte* slot;
n_recs = page_get_n_recs(page);
if (UNIV_UNLIKELY(n_recs > n_dense)) { page_zip_fail(("page_zip_dir_decode 1: %lu > %lu\n", (ulong) n_recs, (ulong) n_dense)); return(FALSE); }
/* Traverse the list of stored records in the sorting order,
starting from the first user record. */
slot = page + (srv_page_size - PAGE_DIR - PAGE_DIR_SLOT_SIZE); UNIV_PREFETCH_RW(slot);
/* Zero out the page trailer. */ memset(slot + PAGE_DIR_SLOT_SIZE, 0, PAGE_DIR);
mach_write_to_2(slot, PAGE_NEW_INFIMUM); slot -= PAGE_DIR_SLOT_SIZE; UNIV_PREFETCH_RW(slot);
/* Initialize the sparse directory and copy the dense directory. */ for (i = 0; i < n_recs; i++) { ulint offs = page_zip_dir_get(page_zip, i);
if (offs & PAGE_ZIP_DIR_SLOT_OWNED) { mach_write_to_2(slot, offs & PAGE_ZIP_DIR_SLOT_MASK); slot -= PAGE_DIR_SLOT_SIZE; UNIV_PREFETCH_RW(slot); }
if (UNIV_UNLIKELY((offs & PAGE_ZIP_DIR_SLOT_MASK) < PAGE_ZIP_START + REC_N_NEW_EXTRA_BYTES)) { page_zip_fail(("page_zip_dir_decode 2: %u %u %lx\n", (unsigned) i, (unsigned) n_recs, (ulong) offs)); return(FALSE); }
recs[i] = page + (offs & PAGE_ZIP_DIR_SLOT_MASK); }
mach_write_to_2(slot, PAGE_NEW_SUPREMUM); { const page_dir_slot_t* last_slot = page_dir_get_nth_slot( page, page_dir_get_n_slots(page) - 1U);
if (UNIV_UNLIKELY(slot != last_slot)) { page_zip_fail(("page_zip_dir_decode 3: %p != %p\n", (const void*) slot, (const void*) last_slot)); return(FALSE); } }
/* Copy the rest of the dense directory. */ for (; i < n_dense; i++) { ulint offs = page_zip_dir_get(page_zip, i);
if (UNIV_UNLIKELY(offs & ~PAGE_ZIP_DIR_SLOT_MASK)) { page_zip_fail(("page_zip_dir_decode 4: %u %u %lx\n", (unsigned) i, (unsigned) n_dense, (ulong) offs)); return(FALSE); }
recs[i] = page + offs; }
std::sort(recs, recs + n_dense); return(TRUE);}
/**********************************************************************//**
Initialize the REC_N_NEW_EXTRA_BYTES of each record.@return TRUE on success, FALSE on failure */staticiboolpage_zip_set_extra_bytes(/*=====================*/ const page_zip_des_t* page_zip,/*!< in: compressed page */ page_t* page, /*!< in/out: uncompressed page */ ulint info_bits)/*!< in: REC_INFO_MIN_REC_FLAG or 0 */{ ulint n; ulint i; ulint n_owned = 1; ulint offs; rec_t* rec;
n = page_get_n_recs(page); rec = page + PAGE_NEW_INFIMUM;
for (i = 0; i < n; i++) { offs = page_zip_dir_get(page_zip, i);
if (offs & PAGE_ZIP_DIR_SLOT_DEL) { info_bits |= REC_INFO_DELETED_FLAG; } if (UNIV_UNLIKELY(offs & PAGE_ZIP_DIR_SLOT_OWNED)) { info_bits |= n_owned; n_owned = 1; } else { n_owned++; } offs &= PAGE_ZIP_DIR_SLOT_MASK; if (UNIV_UNLIKELY(offs < PAGE_ZIP_START + REC_N_NEW_EXTRA_BYTES)) { page_zip_fail(("page_zip_set_extra_bytes 1:" " %u %u %lx\n", (unsigned) i, (unsigned) n, (ulong) offs)); return(FALSE); }
rec_set_next_offs_new(rec, offs); rec = page + offs; rec[-REC_N_NEW_EXTRA_BYTES] = (byte) info_bits; info_bits = 0; }
/* Set the next pointer of the last user record. */ rec_set_next_offs_new(rec, PAGE_NEW_SUPREMUM);
/* Set n_owned of the supremum record. */ page[PAGE_NEW_SUPREMUM - REC_N_NEW_EXTRA_BYTES] = (byte) n_owned;
/* The dense directory excludes the infimum and supremum records. */ n = ulint(page_dir_get_n_heap(page)) - PAGE_HEAP_NO_USER_LOW;
if (i >= n) { if (UNIV_LIKELY(i == n)) { return(TRUE); }
page_zip_fail(("page_zip_set_extra_bytes 2: %u != %u\n", (unsigned) i, (unsigned) n)); return(FALSE); }
offs = page_zip_dir_get(page_zip, i);
/* Set the extra bytes of deleted records on the free list. */ for (;;) { if (UNIV_UNLIKELY(!offs) || UNIV_UNLIKELY(offs & ~PAGE_ZIP_DIR_SLOT_MASK)) {
page_zip_fail(("page_zip_set_extra_bytes 3: %lx\n", (ulong) offs)); return(FALSE); }
rec = page + offs; rec[-REC_N_NEW_EXTRA_BYTES] = 0; /* info_bits and n_owned */
if (++i == n) { break; }
offs = page_zip_dir_get(page_zip, i); rec_set_next_offs_new(rec, offs); }
/* Terminate the free list. */ rec[-REC_N_NEW_EXTRA_BYTES] = 0; /* info_bits and n_owned */ rec_set_next_offs_new(rec, 0);
return(TRUE);}
/**********************************************************************//**
Apply the modification log to a record containing externally storedcolumns. Do not copy the fields that are stored separately.@return pointer to modification log, or NULL on failure */staticconst byte*page_zip_apply_log_ext(/*===================*/ rec_t* rec, /*!< in/out: record */ const offset_t* offsets, /*!< in: rec_get_offsets(rec) */ ulint trx_id_col, /*!< in: position of of DB_TRX_ID */ const byte* data, /*!< in: modification log */ const byte* end) /*!< in: end of modification log */{ ulint i; ulint len; byte* next_out = rec;
/* Check if there are any externally stored columns.
For each externally stored column, skip the BTR_EXTERN_FIELD_REF. */
for (i = 0; i < rec_offs_n_fields(offsets); i++) { byte* dst;
if (UNIV_UNLIKELY(i == trx_id_col)) { /* Skip trx_id and roll_ptr */ dst = rec_get_nth_field(rec, offsets, i, &len); if (UNIV_UNLIKELY(dst - next_out >= end - data) || UNIV_UNLIKELY (len < (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN)) || rec_offs_nth_extern(offsets, i)) { page_zip_fail(("page_zip_apply_log_ext:" " trx_id len %lu," " %p - %p >= %p - %p\n", (ulong) len, (const void*) dst, (const void*) next_out, (const void*) end, (const void*) data)); return(NULL); }
memcpy(next_out, data, ulint(dst - next_out)); data += ulint(dst - next_out); next_out = dst + (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); } else if (rec_offs_nth_extern(offsets, i)) { dst = rec_get_nth_field(rec, offsets, i, &len); ut_ad(len >= BTR_EXTERN_FIELD_REF_SIZE);
len += ulint(dst - next_out) - BTR_EXTERN_FIELD_REF_SIZE;
if (UNIV_UNLIKELY(data + len >= end)) { page_zip_fail(("page_zip_apply_log_ext:" " ext %p+%lu >= %p\n", (const void*) data, (ulong) len, (const void*) end)); return(NULL); }
memcpy(next_out, data, len); data += len; next_out += len + BTR_EXTERN_FIELD_REF_SIZE; } }
/* Copy the last bytes of the record. */ len = ulint(rec_get_end(rec, offsets) - next_out); if (UNIV_UNLIKELY(data + len >= end)) { page_zip_fail(("page_zip_apply_log_ext:" " last %p+%lu >= %p\n", (const void*) data, (ulong) len, (const void*) end)); return(NULL); } memcpy(next_out, data, len); data += len;
return(data);}
/**********************************************************************//**
Apply the modification log to an uncompressed page.Do not copy the fields that are stored separately.@return pointer to end of modification log, or NULL on failure */staticconst byte*page_zip_apply_log(/*===============*/ const byte* data, /*!< in: modification log */ ulint size, /*!< in: maximum length of the log, in bytes */ rec_t** recs, /*!< in: dense page directory,
sorted by address (indexed by heap_no - PAGE_HEAP_NO_USER_LOW) */ ulint n_dense,/*!< in: size of recs[] */ bool is_leaf,/*!< in: whether this is a leaf page */ ulint trx_id_col,/*!< in: column number of trx_id in the index,
or ULINT_UNDEFINED if none */ ulint heap_status, /*!< in: heap_no and status bits for
the next record to uncompress */ dict_index_t* index, /*!< in: index of the page */ offset_t* offsets)/*!< in/out: work area for
rec_get_offsets_reverse() */{ const byte* const end = data + size;
for (;;) { ulint val; rec_t* rec; ulint len; ulint hs;
val = *data++; if (UNIV_UNLIKELY(!val)) { return(data - 1); } if (val & 0x80) { val = (val & 0x7f) << 8 | *data++; if (UNIV_UNLIKELY(!val)) { page_zip_fail(("page_zip_apply_log:" " invalid val %x%x\n", data[-2], data[-1])); return(NULL); } } if (UNIV_UNLIKELY(data >= end)) { page_zip_fail(("page_zip_apply_log: %p >= %p\n", (const void*) data, (const void*) end)); return(NULL); } if (UNIV_UNLIKELY((val >> 1) > n_dense)) { page_zip_fail(("page_zip_apply_log: %lu>>1 > %lu\n", (ulong) val, (ulong) n_dense)); return(NULL); }
/* Determine the heap number and status bits of the record. */ rec = recs[(val >> 1) - 1];
hs = ((val >> 1) + 1) << REC_HEAP_NO_SHIFT; hs |= heap_status & ((1 << REC_HEAP_NO_SHIFT) - 1);
/* This may either be an old record that is being
overwritten (updated in place, or allocated from the free list), or a new record, with the next available_heap_no. */ if (UNIV_UNLIKELY(hs > heap_status)) { page_zip_fail(("page_zip_apply_log: %lu > %lu\n", (ulong) hs, (ulong) heap_status)); return(NULL); } else if (hs == heap_status) { /* A new record was allocated from the heap. */ if (UNIV_UNLIKELY(val & 1)) { /* Only existing records may be cleared. */ page_zip_fail(("page_zip_apply_log:" " attempting to create" " deleted rec %lu\n", (ulong) hs)); return(NULL); } heap_status += 1 << REC_HEAP_NO_SHIFT; }
mach_write_to_2(rec - REC_NEW_HEAP_NO, hs);
if (val & 1) { /* Clear the data bytes of the record. */ mem_heap_t* heap = NULL; offset_t* offs; offs = rec_get_offsets(rec, index, offsets, is_leaf, ULINT_UNDEFINED, &heap); memset(rec, 0, rec_offs_data_size(offs));
if (UNIV_LIKELY_NULL(heap)) { mem_heap_free(heap); } continue; }
compile_time_assert(REC_STATUS_NODE_PTR == TRUE); rec_get_offsets_reverse(data, index, hs & REC_STATUS_NODE_PTR, offsets); /* Silence a debug assertion in rec_offs_make_valid().
This will be overwritten in page_zip_set_extra_bytes(), called by page_zip_decompress_low(). */ ut_d(rec[-REC_NEW_INFO_BITS] = 0); rec_offs_make_valid(rec, index, is_leaf, offsets);
/* Copy the extra bytes (backwards). */ { byte* start = rec_get_start(rec, offsets); byte* b = rec - REC_N_NEW_EXTRA_BYTES; while (b != start) { *--b = *data++; } }
/* Copy the data bytes. */ if (UNIV_UNLIKELY(rec_offs_any_extern(offsets))) { /* Non-leaf nodes should not contain any
externally stored columns. */ if (UNIV_UNLIKELY(hs & REC_STATUS_NODE_PTR)) { page_zip_fail(("page_zip_apply_log:" " %lu&REC_STATUS_NODE_PTR\n", (ulong) hs)); return(NULL); }
data = page_zip_apply_log_ext( rec, offsets, trx_id_col, data, end);
if (UNIV_UNLIKELY(!data)) { return(NULL); } } else if (UNIV_UNLIKELY(hs & REC_STATUS_NODE_PTR)) { len = rec_offs_data_size(offsets) - REC_NODE_PTR_SIZE; /* Copy the data bytes, except node_ptr. */ if (UNIV_UNLIKELY(data + len >= end)) { page_zip_fail(("page_zip_apply_log:" " node_ptr %p+%lu >= %p\n", (const void*) data, (ulong) len, (const void*) end)); return(NULL); } memcpy(rec, data, len); data += len; } else if (UNIV_LIKELY(trx_id_col == ULINT_UNDEFINED)) { len = rec_offs_data_size(offsets);
/* Copy all data bytes of
a record in a secondary index. */ if (UNIV_UNLIKELY(data + len >= end)) { page_zip_fail(("page_zip_apply_log:" " sec %p+%lu >= %p\n", (const void*) data, (ulong) len, (const void*) end)); return(NULL); }
memcpy(rec, data, len); data += len; } else { /* Skip DB_TRX_ID and DB_ROLL_PTR. */ ulint l = rec_get_nth_field_offs(offsets, trx_id_col, &len); byte* b;
if (UNIV_UNLIKELY(data + l >= end) || UNIV_UNLIKELY(len < (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN))) { page_zip_fail(("page_zip_apply_log:" " trx_id %p+%lu >= %p\n", (const void*) data, (ulong) l, (const void*) end)); return(NULL); }
/* Copy any preceding data bytes. */ memcpy(rec, data, l); data += l;
/* Copy any bytes following DB_TRX_ID, DB_ROLL_PTR. */ b = rec + l + (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); len = ulint(rec_get_end(rec, offsets) - b); if (UNIV_UNLIKELY(data + len >= end)) { page_zip_fail(("page_zip_apply_log:" " clust %p+%lu >= %p\n", (const void*) data, (ulong) len, (const void*) end)); return(NULL); } memcpy(b, data, len); data += len; } }}
/**********************************************************************//**
Set the heap_no in a record, and skip the fixed-size record headerthat is not included in the d_stream.@return TRUE on success, FALSE if d_stream does not end at rec */staticiboolpage_zip_decompress_heap_no(/*========================*/ z_stream* d_stream, /*!< in/out: compressed page stream */ rec_t* rec, /*!< in/out: record */ ulint& heap_status) /*!< in/out: heap_no and status bits */{ if (d_stream->next_out != rec - REC_N_NEW_EXTRA_BYTES) { /* n_dense has grown since the page was last compressed. */ return(FALSE); }
/* Skip the REC_N_NEW_EXTRA_BYTES. */ d_stream->next_out = rec;
/* Set heap_no and the status bits. */ mach_write_to_2(rec - REC_NEW_HEAP_NO, heap_status); heap_status += 1 << REC_HEAP_NO_SHIFT; return(TRUE);}
/**********************************************************************//**
Decompress the records of a node pointer page.@return TRUE on success, FALSE on failure */staticiboolpage_zip_decompress_node_ptrs(/*==========================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ z_stream* d_stream, /*!< in/out: compressed page stream */ rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense, /*!< in: size of recs[] */ dict_index_t* index, /*!< in: the index of the page */ offset_t* offsets, /*!< in/out: temporary offsets */ mem_heap_t* heap) /*!< in: temporary memory heap */{ ulint heap_status = REC_STATUS_NODE_PTR | PAGE_HEAP_NO_USER_LOW << REC_HEAP_NO_SHIFT; ulint slot; const byte* storage;
/* Subtract the space reserved for uncompressed data. */ d_stream->avail_in -= static_cast<uInt>( n_dense * (PAGE_ZIP_DIR_SLOT_SIZE + REC_NODE_PTR_SIZE));
/* Decompress the records in heap_no order. */ for (slot = 0; slot < n_dense; slot++) { rec_t* rec = recs[slot];
d_stream->avail_out = static_cast<uInt>( rec - REC_N_NEW_EXTRA_BYTES - d_stream->next_out);
ut_ad(d_stream->avail_out < srv_page_size - PAGE_ZIP_START - PAGE_DIR); switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: page_zip_decompress_heap_no( d_stream, rec, heap_status); goto zlib_done; case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_node_ptrs:" " 1 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; }
if (!page_zip_decompress_heap_no( d_stream, rec, heap_status)) { ut_ad(0); }
/* Read the offsets. The status bits are needed here. */ offsets = rec_get_offsets(rec, index, offsets, false, ULINT_UNDEFINED, &heap);
/* Non-leaf nodes should not have any externally
stored columns. */ ut_ad(!rec_offs_any_extern(offsets));
/* Decompress the data bytes, except node_ptr. */ d_stream->avail_out =static_cast<uInt>( rec_offs_data_size(offsets) - REC_NODE_PTR_SIZE);
switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: goto zlib_done; case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_node_ptrs:" " 2 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; }
/* Clear the node pointer in case the record
will be deleted and the space will be reallocated to a smaller record. */ memset(d_stream->next_out, 0, REC_NODE_PTR_SIZE); d_stream->next_out += REC_NODE_PTR_SIZE;
ut_ad(d_stream->next_out == rec_get_end(rec, offsets)); }
/* Decompress any trailing garbage, in case the last record was
allocated from an originally longer space on the free list. */ d_stream->avail_out = static_cast<uInt>( page_header_get_field(page_zip->data, PAGE_HEAP_TOP) - page_offset(d_stream->next_out)); if (UNIV_UNLIKELY(d_stream->avail_out > srv_page_size - PAGE_ZIP_START - PAGE_DIR)) {
page_zip_fail(("page_zip_decompress_node_ptrs:" " avail_out = %u\n", d_stream->avail_out)); goto zlib_error; }
if (UNIV_UNLIKELY(inflate(d_stream, Z_FINISH) != Z_STREAM_END)) { page_zip_fail(("page_zip_decompress_node_ptrs:" " inflate(Z_FINISH)=%s\n", d_stream->msg));zlib_error: inflateEnd(d_stream); return(FALSE); }
/* Note that d_stream->avail_out > 0 may hold here
if the modification log is nonempty. */
zlib_done: if (UNIV_UNLIKELY(inflateEnd(d_stream) != Z_OK)) { ut_error; }
{ page_t* page = page_align(d_stream->next_out);
/* Clear the unused heap space on the uncompressed page. */ memset(d_stream->next_out, 0, ulint(page_dir_get_nth_slot(page, page_dir_get_n_slots(page) - 1U) - d_stream->next_out)); }
#ifdef UNIV_DEBUG
page_zip->m_start = unsigned(PAGE_DATA + d_stream->total_in);#endif /* UNIV_DEBUG */
/* Apply the modification log. */ { const byte* mod_log_ptr; mod_log_ptr = page_zip_apply_log(d_stream->next_in, d_stream->avail_in + 1, recs, n_dense, false, ULINT_UNDEFINED, heap_status, index, offsets);
if (UNIV_UNLIKELY(!mod_log_ptr)) { return(FALSE); } page_zip->m_end = unsigned(mod_log_ptr - page_zip->data); page_zip->m_nonempty = mod_log_ptr != d_stream->next_in; }
if (UNIV_UNLIKELY (page_zip_get_trailer_len(page_zip, dict_index_is_clust(index)) + page_zip->m_end >= page_zip_get_size(page_zip))) { page_zip_fail(("page_zip_decompress_node_ptrs:" " %lu + %lu >= %lu, %lu\n", (ulong) page_zip_get_trailer_len( page_zip, dict_index_is_clust(index)), (ulong) page_zip->m_end, (ulong) page_zip_get_size(page_zip), (ulong) dict_index_is_clust(index))); return(FALSE); }
/* Restore the uncompressed columns in heap_no order. */ storage = page_zip_dir_start_low(page_zip, n_dense);
for (slot = 0; slot < n_dense; slot++) { rec_t* rec = recs[slot];
offsets = rec_get_offsets(rec, index, offsets, false, ULINT_UNDEFINED, &heap); /* Non-leaf nodes should not have any externally
stored columns. */ ut_ad(!rec_offs_any_extern(offsets)); storage -= REC_NODE_PTR_SIZE;
memcpy(rec_get_end(rec, offsets) - REC_NODE_PTR_SIZE, storage, REC_NODE_PTR_SIZE); }
return(TRUE);}
/**********************************************************************//**
Decompress the records of a leaf node of a secondary index.@return TRUE on success, FALSE on failure */staticiboolpage_zip_decompress_sec(/*====================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ z_stream* d_stream, /*!< in/out: compressed page stream */ rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense, /*!< in: size of recs[] */ dict_index_t* index, /*!< in: the index of the page */ offset_t* offsets) /*!< in/out: temporary offsets */{ ulint heap_status = REC_STATUS_ORDINARY | PAGE_HEAP_NO_USER_LOW << REC_HEAP_NO_SHIFT; ulint slot;
ut_a(!dict_index_is_clust(index));
/* Subtract the space reserved for uncompressed data. */ d_stream->avail_in -= static_cast<uint>( n_dense * PAGE_ZIP_DIR_SLOT_SIZE);
for (slot = 0; slot < n_dense; slot++) { rec_t* rec = recs[slot];
/* Decompress everything up to this record. */ d_stream->avail_out = static_cast<uint>( rec - REC_N_NEW_EXTRA_BYTES - d_stream->next_out);
if (UNIV_LIKELY(d_stream->avail_out)) { switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: page_zip_decompress_heap_no( d_stream, rec, heap_status); goto zlib_done; case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_sec:" " inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; } }
if (!page_zip_decompress_heap_no( d_stream, rec, heap_status)) { ut_ad(0); } }
/* Decompress the data of the last record and any trailing garbage,
in case the last record was allocated from an originally longer space on the free list. */ d_stream->avail_out = static_cast<uInt>( page_header_get_field(page_zip->data, PAGE_HEAP_TOP) - page_offset(d_stream->next_out)); if (UNIV_UNLIKELY(d_stream->avail_out > srv_page_size - PAGE_ZIP_START - PAGE_DIR)) {
page_zip_fail(("page_zip_decompress_sec:" " avail_out = %u\n", d_stream->avail_out)); goto zlib_error; }
if (UNIV_UNLIKELY(inflate(d_stream, Z_FINISH) != Z_STREAM_END)) { page_zip_fail(("page_zip_decompress_sec:" " inflate(Z_FINISH)=%s\n", d_stream->msg));zlib_error: inflateEnd(d_stream); return(FALSE); }
/* Note that d_stream->avail_out > 0 may hold here
if the modification log is nonempty. */
zlib_done: if (UNIV_UNLIKELY(inflateEnd(d_stream) != Z_OK)) { ut_error; }
{ page_t* page = page_align(d_stream->next_out);
/* Clear the unused heap space on the uncompressed page. */ memset(d_stream->next_out, 0, ulint(page_dir_get_nth_slot(page, page_dir_get_n_slots(page) - 1U) - d_stream->next_out)); }
ut_d(page_zip->m_start = unsigned(PAGE_DATA + d_stream->total_in));
/* Apply the modification log. */ { const byte* mod_log_ptr; mod_log_ptr = page_zip_apply_log(d_stream->next_in, d_stream->avail_in + 1, recs, n_dense, true, ULINT_UNDEFINED, heap_status, index, offsets);
if (UNIV_UNLIKELY(!mod_log_ptr)) { return(FALSE); } page_zip->m_end = unsigned(mod_log_ptr - page_zip->data); page_zip->m_nonempty = mod_log_ptr != d_stream->next_in; }
if (UNIV_UNLIKELY(page_zip_get_trailer_len(page_zip, FALSE) + page_zip->m_end >= page_zip_get_size(page_zip))) {
page_zip_fail(("page_zip_decompress_sec: %lu + %lu >= %lu\n", (ulong) page_zip_get_trailer_len( page_zip, FALSE), (ulong) page_zip->m_end, (ulong) page_zip_get_size(page_zip))); return(FALSE); }
/* There are no uncompressed columns on leaf pages of
secondary indexes. */
return(TRUE);}
/**********************************************************************//**
Decompress a record of a leaf node of a clustered index that containsexternally stored columns.@return TRUE on success */staticiboolpage_zip_decompress_clust_ext(/*==========================*/ z_stream* d_stream, /*!< in/out: compressed page stream */ rec_t* rec, /*!< in/out: record */ const offset_t* offsets, /*!< in: rec_get_offsets(rec) */ ulint trx_id_col) /*!< in: position of of DB_TRX_ID */{ ulint i;
for (i = 0; i < rec_offs_n_fields(offsets); i++) { ulint len; byte* dst;
if (UNIV_UNLIKELY(i == trx_id_col)) { /* Skip trx_id and roll_ptr */ dst = rec_get_nth_field(rec, offsets, i, &len); if (UNIV_UNLIKELY(len < DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN)) {
page_zip_fail(("page_zip_decompress_clust_ext:" " len[%lu] = %lu\n", (ulong) i, (ulong) len)); return(FALSE); }
if (rec_offs_nth_extern(offsets, i)) {
page_zip_fail(("page_zip_decompress_clust_ext:" " DB_TRX_ID at %lu is ext\n", (ulong) i)); return(FALSE); }
d_stream->avail_out = static_cast<uInt>( dst - d_stream->next_out);
switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_clust_ext:" " 1 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); return(FALSE); }
ut_ad(d_stream->next_out == dst);
/* Clear DB_TRX_ID and DB_ROLL_PTR in order to
avoid uninitialized bytes in case the record is affected by page_zip_apply_log(). */ memset(dst, 0, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
d_stream->next_out += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN; } else if (rec_offs_nth_extern(offsets, i)) { dst = rec_get_nth_field(rec, offsets, i, &len); ut_ad(len >= BTR_EXTERN_FIELD_REF_SIZE); dst += len - BTR_EXTERN_FIELD_REF_SIZE;
d_stream->avail_out = static_cast<uInt>( dst - d_stream->next_out); switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_clust_ext:" " 2 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); return(FALSE); }
ut_ad(d_stream->next_out == dst);
/* Clear the BLOB pointer in case
the record will be deleted and the space will not be reused. Note that the final initialization of the BLOB pointers (copying from "externs" or clearing) will have to take place only after the page modification log has been applied. Otherwise, we could end up with an uninitialized BLOB pointer when a record is deleted, reallocated and deleted. */ memset(d_stream->next_out, 0, BTR_EXTERN_FIELD_REF_SIZE); d_stream->next_out += BTR_EXTERN_FIELD_REF_SIZE; } }
return(TRUE);}
/**********************************************************************//**
Compress the records of a leaf node of a clustered index.@return TRUE on success, FALSE on failure */staticiboolpage_zip_decompress_clust(/*======================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ z_stream* d_stream, /*!< in/out: compressed page stream */ rec_t** recs, /*!< in: dense page directory
sorted by address */ ulint n_dense, /*!< in: size of recs[] */ dict_index_t* index, /*!< in: the index of the page */ ulint trx_id_col, /*!< index of the trx_id column */ offset_t* offsets, /*!< in/out: temporary offsets */ mem_heap_t* heap) /*!< in: temporary memory heap */{ int err; ulint slot; ulint heap_status = REC_STATUS_ORDINARY | PAGE_HEAP_NO_USER_LOW << REC_HEAP_NO_SHIFT; const byte* storage; const byte* externs;
ut_a(dict_index_is_clust(index));
/* Subtract the space reserved for uncompressed data. */ d_stream->avail_in -= static_cast<uInt>(n_dense) * (PAGE_ZIP_CLUST_LEAF_SLOT_SIZE);
/* Decompress the records in heap_no order. */ for (slot = 0; slot < n_dense; slot++) { rec_t* rec = recs[slot];
d_stream->avail_out =static_cast<uInt>( rec - REC_N_NEW_EXTRA_BYTES - d_stream->next_out);
ut_ad(d_stream->avail_out < srv_page_size - PAGE_ZIP_START - PAGE_DIR); err = inflate(d_stream, Z_SYNC_FLUSH); switch (err) { case Z_STREAM_END: page_zip_decompress_heap_no( d_stream, rec, heap_status); goto zlib_done; case Z_OK: case Z_BUF_ERROR: if (UNIV_LIKELY(!d_stream->avail_out)) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_clust:" " 1 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; }
if (!page_zip_decompress_heap_no( d_stream, rec, heap_status)) { ut_ad(0); }
/* Read the offsets. The status bits are needed here. */ offsets = rec_get_offsets(rec, index, offsets, true, ULINT_UNDEFINED, &heap);
/* This is a leaf page in a clustered index. */
/* Check if there are any externally stored columns.
For each externally stored column, restore the BTR_EXTERN_FIELD_REF separately. */
if (rec_offs_any_extern(offsets)) { if (UNIV_UNLIKELY (!page_zip_decompress_clust_ext( d_stream, rec, offsets, trx_id_col))) {
goto zlib_error; } } else { /* Skip trx_id and roll_ptr */ ulint len; byte* dst = rec_get_nth_field(rec, offsets, trx_id_col, &len); if (UNIV_UNLIKELY(len < DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN)) {
page_zip_fail(("page_zip_decompress_clust:" " len = %lu\n", (ulong) len)); goto zlib_error; }
d_stream->avail_out = static_cast<uInt>( dst - d_stream->next_out);
switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_clust:" " 2 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; }
ut_ad(d_stream->next_out == dst);
/* Clear DB_TRX_ID and DB_ROLL_PTR in order to
avoid uninitialized bytes in case the record is affected by page_zip_apply_log(). */ memset(dst, 0, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
d_stream->next_out += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN; }
/* Decompress the last bytes of the record. */ d_stream->avail_out = static_cast<uInt>( rec_get_end(rec, offsets) - d_stream->next_out);
switch (inflate(d_stream, Z_SYNC_FLUSH)) { case Z_STREAM_END: case Z_OK: case Z_BUF_ERROR: if (!d_stream->avail_out) { break; } /* fall through */ default: page_zip_fail(("page_zip_decompress_clust:" " 3 inflate(Z_SYNC_FLUSH)=%s\n", d_stream->msg)); goto zlib_error; } }
/* Decompress any trailing garbage, in case the last record was
allocated from an originally longer space on the free list. */ d_stream->avail_out = static_cast<uInt>( page_header_get_field(page_zip->data, PAGE_HEAP_TOP) - page_offset(d_stream->next_out)); if (UNIV_UNLIKELY(d_stream->avail_out > srv_page_size - PAGE_ZIP_START - PAGE_DIR)) {
page_zip_fail(("page_zip_decompress_clust:" " avail_out = %u\n", d_stream->avail_out)); goto zlib_error; }
if (UNIV_UNLIKELY(inflate(d_stream, Z_FINISH) != Z_STREAM_END)) { page_zip_fail(("page_zip_decompress_clust:" " inflate(Z_FINISH)=%s\n", d_stream->msg));zlib_error: inflateEnd(d_stream); return(FALSE); }
/* Note that d_stream->avail_out > 0 may hold here
if the modification log is nonempty. */
zlib_done: if (UNIV_UNLIKELY(inflateEnd(d_stream) != Z_OK)) { ut_error; }
{ page_t* page = page_align(d_stream->next_out);
/* Clear the unused heap space on the uncompressed page. */ memset(d_stream->next_out, 0, ulint(page_dir_get_nth_slot(page, page_dir_get_n_slots(page) - 1U) - d_stream->next_out)); }
ut_d(page_zip->m_start = unsigned(PAGE_DATA + d_stream->total_in));
/* Apply the modification log. */ { const byte* mod_log_ptr; mod_log_ptr = page_zip_apply_log(d_stream->next_in, d_stream->avail_in + 1, recs, n_dense, true, trx_id_col, heap_status, index, offsets);
if (UNIV_UNLIKELY(!mod_log_ptr)) { return(FALSE); } page_zip->m_end = unsigned(mod_log_ptr - page_zip->data); page_zip->m_nonempty = mod_log_ptr != d_stream->next_in; }
if (UNIV_UNLIKELY(page_zip_get_trailer_len(page_zip, TRUE) + page_zip->m_end >= page_zip_get_size(page_zip))) {
page_zip_fail(("page_zip_decompress_clust: %lu + %lu >= %lu\n", (ulong) page_zip_get_trailer_len( page_zip, TRUE), (ulong) page_zip->m_end, (ulong) page_zip_get_size(page_zip))); return(FALSE); }
storage = page_zip_dir_start_low(page_zip, n_dense);
externs = storage - n_dense * (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
/* Restore the uncompressed columns in heap_no order. */
for (slot = 0; slot < n_dense; slot++) { ulint i; ulint len; byte* dst; rec_t* rec = recs[slot]; bool exists = !page_zip_dir_find_free( page_zip, page_offset(rec)); offsets = rec_get_offsets(rec, index, offsets, true, ULINT_UNDEFINED, &heap);
dst = rec_get_nth_field(rec, offsets, trx_id_col, &len); ut_ad(len >= DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); storage -= DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN; memcpy(dst, storage, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
/* Check if there are any externally stored
columns in this record. For each externally stored column, restore or clear the BTR_EXTERN_FIELD_REF. */ if (!rec_offs_any_extern(offsets)) { continue; }
for (i = 0; i < rec_offs_n_fields(offsets); i++) { if (!rec_offs_nth_extern(offsets, i)) { continue; } dst = rec_get_nth_field(rec, offsets, i, &len);
if (UNIV_UNLIKELY(len < BTR_EXTERN_FIELD_REF_SIZE)) { page_zip_fail(("page_zip_decompress_clust:" " %lu < 20\n", (ulong) len)); return(FALSE); }
dst += len - BTR_EXTERN_FIELD_REF_SIZE;
if (UNIV_LIKELY(exists)) { /* Existing record:
restore the BLOB pointer */ externs -= BTR_EXTERN_FIELD_REF_SIZE;
if (UNIV_UNLIKELY (externs < page_zip->data + page_zip->m_end)) { page_zip_fail(("page_zip_" "decompress_clust:" " %p < %p + %lu\n", (const void*) externs, (const void*) page_zip->data, (ulong) page_zip->m_end)); return(FALSE); }
memcpy(dst, externs, BTR_EXTERN_FIELD_REF_SIZE);
page_zip->n_blobs++; } else { /* Deleted record:
clear the BLOB pointer */ memset(dst, 0, BTR_EXTERN_FIELD_REF_SIZE); } } }
return(TRUE);}
/**********************************************************************//**
Decompress a page. This function should tolerate errors on the compressedpage. Instead of letting assertions fail, it will return FALSE if aninconsistency is detected.@return TRUE on success, FALSE on failure */staticiboolpage_zip_decompress_low(/*====================*/ page_zip_des_t* page_zip,/*!< in: data, ssize;
out: m_start, m_end, m_nonempty, n_blobs */ page_t* page, /*!< out: uncompressed page, may be trashed */ ibool all) /*!< in: TRUE=decompress the whole page;
FALSE=verify but do not copy some page header fields that should not change after page creation */{ z_stream d_stream; dict_index_t* index = NULL; rec_t** recs; /*!< dense page directory, sorted by address */ ulint n_dense;/* number of user records on the page */ ulint trx_id_col = ULINT_UNDEFINED; mem_heap_t* heap; offset_t* offsets;
ut_ad(page_zip_simple_validate(page_zip)); UNIV_MEM_ASSERT_W(page, srv_page_size); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
/* The dense directory excludes the infimum and supremum records. */ n_dense = page_dir_get_n_heap(page_zip->data) - PAGE_HEAP_NO_USER_LOW; if (UNIV_UNLIKELY(n_dense * PAGE_ZIP_DIR_SLOT_SIZE >= page_zip_get_size(page_zip))) { page_zip_fail(("page_zip_decompress 1: %lu %lu\n", (ulong) n_dense, (ulong) page_zip_get_size(page_zip))); return(FALSE); }
heap = mem_heap_create(n_dense * (3 * sizeof *recs) + srv_page_size);
recs = static_cast<rec_t**>( mem_heap_alloc(heap, n_dense * sizeof *recs));
if (all) { /* Copy the page header. */ memcpy_aligned<2>(page, page_zip->data, PAGE_DATA); } else { /* Check that the bytes that we skip are identical. */#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
ut_a(!memcmp(FIL_PAGE_TYPE + page, FIL_PAGE_TYPE + page_zip->data, PAGE_HEADER - FIL_PAGE_TYPE)); ut_a(!memcmp(PAGE_HEADER + PAGE_LEVEL + page, PAGE_HEADER + PAGE_LEVEL + page_zip->data, PAGE_DATA - (PAGE_HEADER + PAGE_LEVEL)));#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
/* Copy the mutable parts of the page header. */ memcpy_aligned<8>(page, page_zip->data, FIL_PAGE_TYPE); memcpy_aligned<2>(PAGE_HEADER + page, PAGE_HEADER + page_zip->data, PAGE_LEVEL - PAGE_N_DIR_SLOTS);
#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
/* Check that the page headers match after copying. */ ut_a(!memcmp(page, page_zip->data, PAGE_DATA));#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
}
#ifdef UNIV_ZIP_DEBUG
/* Clear the uncompressed page, except the header. */ memset(PAGE_DATA + page, 0x55, srv_page_size - PAGE_DATA);#endif /* UNIV_ZIP_DEBUG */
UNIV_MEM_INVALID(PAGE_DATA + page, srv_page_size - PAGE_DATA);
/* Copy the page directory. */ if (UNIV_UNLIKELY(!page_zip_dir_decode(page_zip, page, recs, n_dense))) {zlib_error: mem_heap_free(heap); return(FALSE); }
/* Copy the infimum and supremum records. */ memcpy(page + (PAGE_NEW_INFIMUM - REC_N_NEW_EXTRA_BYTES), infimum_extra, sizeof infimum_extra); if (page_is_empty(page)) { rec_set_next_offs_new(page + PAGE_NEW_INFIMUM, PAGE_NEW_SUPREMUM); } else { rec_set_next_offs_new(page + PAGE_NEW_INFIMUM, page_zip_dir_get(page_zip, 0) & PAGE_ZIP_DIR_SLOT_MASK); } memcpy(page + PAGE_NEW_INFIMUM, infimum_data, sizeof infimum_data); memcpy_aligned<4>(PAGE_NEW_SUPREMUM - REC_N_NEW_EXTRA_BYTES + 1 + page, supremum_extra_data, sizeof supremum_extra_data);
page_zip_set_alloc(&d_stream, heap);
d_stream.next_in = page_zip->data + PAGE_DATA; /* Subtract the space reserved for
the page header and the end marker of the modification log. */ d_stream.avail_in = static_cast<uInt>( page_zip_get_size(page_zip) - (PAGE_DATA + 1)); d_stream.next_out = page + PAGE_ZIP_START; d_stream.avail_out = uInt(srv_page_size - PAGE_ZIP_START);
if (UNIV_UNLIKELY(inflateInit2(&d_stream, srv_page_size_shift) != Z_OK)) { ut_error; }
/* Decode the zlib header and the index information. */ if (UNIV_UNLIKELY(inflate(&d_stream, Z_BLOCK) != Z_OK)) {
page_zip_fail(("page_zip_decompress:" " 1 inflate(Z_BLOCK)=%s\n", d_stream.msg)); goto zlib_error; }
if (UNIV_UNLIKELY(inflate(&d_stream, Z_BLOCK) != Z_OK)) {
page_zip_fail(("page_zip_decompress:" " 2 inflate(Z_BLOCK)=%s\n", d_stream.msg)); goto zlib_error; }
index = page_zip_fields_decode( page + PAGE_ZIP_START, d_stream.next_out, page_is_leaf(page) ? &trx_id_col : NULL, fil_page_get_type(page) == FIL_PAGE_RTREE);
if (UNIV_UNLIKELY(!index)) {
goto zlib_error; }
/* Decompress the user records. */ page_zip->n_blobs = 0; d_stream.next_out = page + PAGE_ZIP_START;
{ /* Pre-allocate the offsets for rec_get_offsets_reverse(). */ ulint n = 1 + 1/* node ptr */ + REC_OFFS_HEADER_SIZE + dict_index_get_n_fields(index);
offsets = static_cast<offset_t*>( mem_heap_alloc(heap, n * sizeof(ulint)));
rec_offs_set_n_alloc(offsets, n); }
/* Decompress the records in heap_no order. */ if (!page_is_leaf(page)) { /* This is a node pointer page. */ ulint info_bits;
if (UNIV_UNLIKELY (!page_zip_decompress_node_ptrs(page_zip, &d_stream, recs, n_dense, index, offsets, heap))) { goto err_exit; }
info_bits = page_has_prev(page) ? 0 : REC_INFO_MIN_REC_FLAG;
if (UNIV_UNLIKELY(!page_zip_set_extra_bytes(page_zip, page, info_bits))) { goto err_exit; } } else if (UNIV_LIKELY(trx_id_col == ULINT_UNDEFINED)) { /* This is a leaf page in a secondary index. */ if (UNIV_UNLIKELY(!page_zip_decompress_sec(page_zip, &d_stream, recs, n_dense, index, offsets))) { goto err_exit; }
if (UNIV_UNLIKELY(!page_zip_set_extra_bytes(page_zip, page, 0))) {err_exit: page_zip_fields_free(index); mem_heap_free(heap); return(FALSE); } } else { /* This is a leaf page in a clustered index. */ if (UNIV_UNLIKELY(!page_zip_decompress_clust(page_zip, &d_stream, recs, n_dense, index, trx_id_col, offsets, heap))) { goto err_exit; }
if (UNIV_UNLIKELY(!page_zip_set_extra_bytes(page_zip, page, 0))) { goto err_exit; } }
ut_a(page_is_comp(page)); UNIV_MEM_ASSERT_RW(page, srv_page_size);
page_zip_fields_free(index); mem_heap_free(heap);
return(TRUE);}
/**********************************************************************//**
Decompress a page. This function should tolerate errors on the compressedpage. Instead of letting assertions fail, it will return FALSE if aninconsistency is detected.@return TRUE on success, FALSE on failure */iboolpage_zip_decompress(/*================*/ page_zip_des_t* page_zip,/*!< in: data, ssize;
out: m_start, m_end, m_nonempty, n_blobs */ page_t* page, /*!< out: uncompressed page, may be trashed */ ibool all) /*!< in: TRUE=decompress the whole page;
FALSE=verify but do not copy some page header fields that should not change after page creation */{ const ulonglong ns = my_interval_timer();
if (!page_zip_decompress_low(page_zip, page, all)) { return(FALSE); }
const uint64_t time_diff = (my_interval_timer() - ns) / 1000; page_zip_stat[page_zip->ssize - 1].decompressed++; page_zip_stat[page_zip->ssize - 1].decompressed_usec += time_diff;
index_id_t index_id = btr_page_get_index_id(page);
if (srv_cmp_per_index_enabled) { mutex_enter(&page_zip_stat_per_index_mutex); page_zip_stat_per_index[index_id].decompressed++; page_zip_stat_per_index[index_id].decompressed_usec += time_diff; mutex_exit(&page_zip_stat_per_index_mutex); }
/* Update the stat counter for LRU policy. */ buf_LRU_stat_inc_unzip();
MONITOR_INC(MONITOR_PAGE_DECOMPRESS);
return(TRUE);}
#ifdef UNIV_ZIP_DEBUG
/**********************************************************************//**
Dump a block of memory on the standard error stream. */staticvoidpage_zip_hexdump_func(/*==================*/ const char* name, /*!< in: name of the data structure */ const void* buf, /*!< in: data */ ulint size) /*!< in: length of the data, in bytes */{ const byte* s = static_cast<const byte*>(buf); ulint addr; const ulint width = 32; /* bytes per line */
fprintf(stderr, "%s:\n", name);
for (addr = 0; addr < size; addr += width) { ulint i;
fprintf(stderr, "%04lx ", (ulong) addr);
i = ut_min(width, size - addr);
while (i--) { fprintf(stderr, "%02x", *s++); }
putc('\n', stderr); }}
/** Dump a block of memory on the standard error stream.
@param buf in: data@param size in: length of the data, in bytes */#define page_zip_hexdump(buf, size) page_zip_hexdump_func(#buf, buf, size)
/** Flag: make page_zip_validate() compare page headers only */bool page_zip_validate_header_only;
/**********************************************************************//**
Check that the compressed and decompressed pages match.@return TRUE if valid, FALSE if not */iboolpage_zip_validate_low(/*==================*/ const page_zip_des_t* page_zip,/*!< in: compressed page */ const page_t* page, /*!< in: uncompressed page */ const dict_index_t* index, /*!< in: index of the page, if known */ ibool sloppy) /*!< in: FALSE=strict,
TRUE=ignore the MIN_REC_FLAG */{ page_zip_des_t temp_page_zip; ibool valid;
if (memcmp(page_zip->data + FIL_PAGE_PREV, page + FIL_PAGE_PREV, FIL_PAGE_LSN - FIL_PAGE_PREV) || memcmp(page_zip->data + FIL_PAGE_TYPE, page + FIL_PAGE_TYPE, 2) || memcmp(page_zip->data + FIL_PAGE_DATA, page + FIL_PAGE_DATA, PAGE_DATA - FIL_PAGE_DATA)) { page_zip_fail(("page_zip_validate: page header\n")); page_zip_hexdump(page_zip, sizeof *page_zip); page_zip_hexdump(page_zip->data, page_zip_get_size(page_zip)); page_zip_hexdump(page, srv_page_size); return(FALSE); }
ut_a(page_is_comp(page));
if (page_zip_validate_header_only) { return(TRUE); }
/* page_zip_decompress() expects the uncompressed page to be
srv_page_size aligned. */ page_t* temp_page = static_cast<byte*>(aligned_malloc(srv_page_size, srv_page_size));
UNIV_MEM_ASSERT_RW(page, srv_page_size); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
temp_page_zip = *page_zip; valid = page_zip_decompress_low(&temp_page_zip, temp_page, TRUE); if (!valid) { fputs("page_zip_validate(): failed to decompress\n", stderr); goto func_exit; } if (page_zip->n_blobs != temp_page_zip.n_blobs) { page_zip_fail(("page_zip_validate: n_blobs: %u!=%u\n", page_zip->n_blobs, temp_page_zip.n_blobs)); valid = FALSE; }#ifdef UNIV_DEBUG
if (page_zip->m_start != temp_page_zip.m_start) { page_zip_fail(("page_zip_validate: m_start: %u!=%u\n", page_zip->m_start, temp_page_zip.m_start)); valid = FALSE; }#endif /* UNIV_DEBUG */
if (page_zip->m_end != temp_page_zip.m_end) { page_zip_fail(("page_zip_validate: m_end: %u!=%u\n", page_zip->m_end, temp_page_zip.m_end)); valid = FALSE; } if (page_zip->m_nonempty != temp_page_zip.m_nonempty) { page_zip_fail(("page_zip_validate(): m_nonempty: %u!=%u\n", page_zip->m_nonempty, temp_page_zip.m_nonempty)); valid = FALSE; } if (memcmp(page + PAGE_HEADER, temp_page + PAGE_HEADER, srv_page_size - PAGE_HEADER - FIL_PAGE_DATA_END)) {
/* In crash recovery, the "minimum record" flag may be
set incorrectly until the mini-transaction is committed. Let us tolerate that difference when we are performing a sloppy validation. */
offset_t* offsets; mem_heap_t* heap; const rec_t* rec; const rec_t* trec; byte info_bits_diff; ulint offset = rec_get_next_offs(page + PAGE_NEW_INFIMUM, TRUE); ut_a(offset >= PAGE_NEW_SUPREMUM); offset -= 5/*REC_NEW_INFO_BITS*/;
info_bits_diff = page[offset] ^ temp_page[offset];
if (info_bits_diff == REC_INFO_MIN_REC_FLAG) { temp_page[offset] = page[offset];
if (!memcmp(page + PAGE_HEADER, temp_page + PAGE_HEADER, srv_page_size - PAGE_HEADER - FIL_PAGE_DATA_END)) {
/* Only the minimum record flag
differed. Let us ignore it. */ page_zip_fail(("page_zip_validate:" " min_rec_flag" " (%s%lu,%lu,0x%02lx)\n", sloppy ? "ignored, " : "", page_get_space_id(page), page_get_page_no(page), (ulong) page[offset])); /* We don't check for spatial index, since
the "minimum record" could be deleted when doing rtr_update_mbr_field. GIS_FIXME: need to validate why rtr_update_mbr_field.() could affect this */ if (index && dict_index_is_spatial(index)) { valid = true; } else { valid = sloppy; } goto func_exit; } }
/* Compare the pointers in the PAGE_FREE list. */ rec = page_header_get_ptr(page, PAGE_FREE); trec = page_header_get_ptr(temp_page, PAGE_FREE);
while (rec || trec) { if (page_offset(rec) != page_offset(trec)) { page_zip_fail(("page_zip_validate:" " PAGE_FREE list: %u!=%u\n", (unsigned) page_offset(rec), (unsigned) page_offset(trec))); valid = FALSE; goto func_exit; }
rec = page_rec_get_next_low(rec, TRUE); trec = page_rec_get_next_low(trec, TRUE); }
/* Compare the records. */ heap = NULL; offsets = NULL; rec = page_rec_get_next_low( page + PAGE_NEW_INFIMUM, TRUE); trec = page_rec_get_next_low( temp_page + PAGE_NEW_INFIMUM, TRUE); const bool is_leaf = page_is_leaf(page);
do { if (page_offset(rec) != page_offset(trec)) { page_zip_fail(("page_zip_validate:" " record list: 0x%02x!=0x%02x\n", (unsigned) page_offset(rec), (unsigned) page_offset(trec))); valid = FALSE; break; }
if (index) { /* Compare the data. */ offsets = rec_get_offsets( rec, index, offsets, is_leaf, ULINT_UNDEFINED, &heap);
if (memcmp(rec - rec_offs_extra_size(offsets), trec - rec_offs_extra_size(offsets), rec_offs_size(offsets))) { page_zip_fail( ("page_zip_validate:" " record content: 0x%02x", (unsigned) page_offset(rec))); valid = FALSE; break; } }
rec = page_rec_get_next_low(rec, TRUE); trec = page_rec_get_next_low(trec, TRUE); } while (rec || trec);
if (heap) { mem_heap_free(heap); } }
func_exit: if (!valid) { page_zip_hexdump(page_zip, sizeof *page_zip); page_zip_hexdump(page_zip->data, page_zip_get_size(page_zip)); page_zip_hexdump(page, srv_page_size); page_zip_hexdump(temp_page, srv_page_size); } aligned_free(temp_page); return(valid);}
/**********************************************************************//**
Check that the compressed and decompressed pages match.@return TRUE if valid, FALSE if not */iboolpage_zip_validate(/*==============*/ const page_zip_des_t* page_zip,/*!< in: compressed page */ const page_t* page, /*!< in: uncompressed page */ const dict_index_t* index) /*!< in: index of the page, if known */{ return(page_zip_validate_low(page_zip, page, index, recv_recovery_is_on()));}#endif /* UNIV_ZIP_DEBUG */
#ifdef UNIV_DEBUG
/**********************************************************************//**
Assert that the compressed and decompressed page headers match.@return TRUE */staticiboolpage_zip_header_cmp(/*================*/ const page_zip_des_t* page_zip,/*!< in: compressed page */ const byte* page) /*!< in: uncompressed page */{ ut_ad(!memcmp(page_zip->data + FIL_PAGE_PREV, page + FIL_PAGE_PREV, FIL_PAGE_LSN - FIL_PAGE_PREV)); ut_ad(!memcmp(page_zip->data + FIL_PAGE_TYPE, page + FIL_PAGE_TYPE, 2)); ut_ad(!memcmp(page_zip->data + FIL_PAGE_DATA, page + FIL_PAGE_DATA, PAGE_DATA - FIL_PAGE_DATA));
return(TRUE);}#endif /* UNIV_DEBUG */
/**********************************************************************//**
Write a record on the compressed page that contains externally storedcolumns. The data must already have been written to the uncompressed page.@return end of modification log */staticbyte*page_zip_write_rec_ext(/*===================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ const page_t* page, /*!< in: page containing rec */ const byte* rec, /*!< in: record being written */ dict_index_t* index, /*!< in: record descriptor */ const offset_t* offsets, /*!< in: rec_get_offsets(rec, index) */ ulint create, /*!< in: nonzero=insert, zero=update */ ulint trx_id_col, /*!< in: position of DB_TRX_ID */ ulint heap_no, /*!< in: heap number of rec */ byte* storage, /*!< in: end of dense page directory */ byte* data) /*!< in: end of modification log */{ const byte* start = rec; ulint i; ulint len; byte* externs = storage; ulint n_ext = rec_offs_n_extern(offsets);
ut_ad(rec_offs_validate(rec, index, offsets)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
externs -= (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) * (page_dir_get_n_heap(page) - PAGE_HEAP_NO_USER_LOW);
/* Note that this will not take into account
the BLOB columns of rec if create==TRUE. */ ut_ad(data + rec_offs_data_size(offsets) - (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) - n_ext * BTR_EXTERN_FIELD_REF_SIZE < externs - BTR_EXTERN_FIELD_REF_SIZE * page_zip->n_blobs);
{ ulint blob_no = page_zip_get_n_prev_extern( page_zip, rec, index); byte* ext_end = externs - page_zip->n_blobs * BTR_EXTERN_FIELD_REF_SIZE; ut_ad(blob_no <= page_zip->n_blobs); externs -= blob_no * BTR_EXTERN_FIELD_REF_SIZE;
if (create) { page_zip->n_blobs += static_cast<unsigned>(n_ext); ASSERT_ZERO_BLOB(ext_end - n_ext * BTR_EXTERN_FIELD_REF_SIZE); memmove(ext_end - n_ext * BTR_EXTERN_FIELD_REF_SIZE, ext_end, ulint(externs - ext_end)); }
ut_a(blob_no + n_ext <= page_zip->n_blobs); }
for (i = 0; i < rec_offs_n_fields(offsets); i++) { const byte* src;
if (UNIV_UNLIKELY(i == trx_id_col)) { ut_ad(!rec_offs_nth_extern(offsets, i)); ut_ad(!rec_offs_nth_extern(offsets, i + 1)); /* Locate trx_id and roll_ptr. */ src = rec_get_nth_field(rec, offsets, i, &len); ut_ad(len == DATA_TRX_ID_LEN); ut_ad(src + DATA_TRX_ID_LEN == rec_get_nth_field( rec, offsets, i + 1, &len)); ut_ad(len == DATA_ROLL_PTR_LEN);
/* Log the preceding fields. */ ASSERT_ZERO(data, src - start); memcpy(data, start, ulint(src - start)); data += src - start; start = src + (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
/* Store trx_id and roll_ptr. */ memcpy(storage - (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) * (heap_no - 1), src, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); i++; /* skip also roll_ptr */ } else if (rec_offs_nth_extern(offsets, i)) { src = rec_get_nth_field(rec, offsets, i, &len);
ut_ad(dict_index_is_clust(index)); ut_ad(len >= BTR_EXTERN_FIELD_REF_SIZE); src += len - BTR_EXTERN_FIELD_REF_SIZE;
ASSERT_ZERO(data, src - start); memcpy(data, start, ulint(src - start)); data += src - start; start = src + BTR_EXTERN_FIELD_REF_SIZE;
/* Store the BLOB pointer. */ externs -= BTR_EXTERN_FIELD_REF_SIZE; ut_ad(data < externs); memcpy(externs, src, BTR_EXTERN_FIELD_REF_SIZE); } }
/* Log the last bytes of the record. */ len = rec_offs_data_size(offsets) - ulint(start - rec);
ASSERT_ZERO(data, len); memcpy(data, start, len); data += len;
return(data);}
/**********************************************************************//**
Write an entire record on the compressed page. The data must alreadyhave been written to the uncompressed page. */voidpage_zip_write_rec(/*===============*/ page_zip_des_t* page_zip,/*!< in/out: compressed page */ const byte* rec, /*!< in: record being written */ dict_index_t* index, /*!< in: the index the record belongs to */ const offset_t* offsets,/*!< in: rec_get_offsets(rec, index) */ ulint create) /*!< in: nonzero=insert, zero=update */{ const page_t* page; byte* data; byte* storage; ulint heap_no; byte* slot;
ut_ad(page_zip_simple_validate(page_zip)); ut_ad(page_zip_get_size(page_zip) > PAGE_DATA + page_zip_dir_size(page_zip)); ut_ad(rec_offs_comp(offsets)); ut_ad(rec_offs_validate(rec, index, offsets));
ut_ad(page_zip->m_start >= PAGE_DATA);
page = page_align(rec);
ut_ad(page_zip_header_cmp(page_zip, page)); ut_ad(page_simple_validate_new((page_t*) page));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
slot = page_zip_dir_find(page_zip, page_offset(rec)); ut_a(slot); /* Copy the delete mark. */ if (rec_get_deleted_flag(rec, TRUE)) { /* In delete-marked records, DB_TRX_ID must
always refer to an existing undo log record. On non-leaf pages, the delete-mark flag is garbage. */ ut_ad(!index->is_primary() || !page_is_leaf(page) || row_get_rec_trx_id(rec, index, offsets)); *slot |= PAGE_ZIP_DIR_SLOT_DEL >> 8; } else { *slot &= ~(PAGE_ZIP_DIR_SLOT_DEL >> 8); }
ut_ad(rec_get_start((rec_t*) rec, offsets) >= page + PAGE_ZIP_START); ut_ad(rec_get_end((rec_t*) rec, offsets) <= page + srv_page_size - PAGE_DIR - PAGE_DIR_SLOT_SIZE * page_dir_get_n_slots(page));
heap_no = rec_get_heap_no_new(rec); ut_ad(heap_no >= PAGE_HEAP_NO_USER_LOW); /* not infimum or supremum */ ut_ad(heap_no < page_dir_get_n_heap(page));
/* Append to the modification log. */ data = page_zip->data + page_zip->m_end; ut_ad(!*data);
/* Identify the record by writing its heap number - 1.
0 is reserved to indicate the end of the modification log. */
if (UNIV_UNLIKELY(heap_no - 1 >= 64)) { *data++ = (byte) (0x80 | (heap_no - 1) >> 7); ut_ad(!*data); } *data++ = (byte) ((heap_no - 1) << 1); ut_ad(!*data);
{ const byte* start = rec - rec_offs_extra_size(offsets); const byte* b = rec - REC_N_NEW_EXTRA_BYTES;
/* Write the extra bytes backwards, so that
rec_offs_extra_size() can be easily computed in page_zip_apply_log() by invoking rec_get_offsets_reverse(). */
while (b != start) { *data++ = *--b; ut_ad(!*data); } }
/* Write the data bytes. Store the uncompressed bytes separately. */ storage = page_zip_dir_start(page_zip);
if (page_is_leaf(page)) { ulint len;
if (dict_index_is_clust(index)) { /* Store separately trx_id, roll_ptr and
the BTR_EXTERN_FIELD_REF of each BLOB column. */ if (rec_offs_any_extern(offsets)) { data = page_zip_write_rec_ext( page_zip, page, rec, index, offsets, create, index->db_trx_id(), heap_no, storage, data); } else { /* Locate trx_id and roll_ptr. */ const byte* src = rec_get_nth_field(rec, offsets, index->db_trx_id(), &len); ut_ad(len == DATA_TRX_ID_LEN); ut_ad(src + DATA_TRX_ID_LEN == rec_get_nth_field( rec, offsets, index->db_roll_ptr(), &len)); ut_ad(len == DATA_ROLL_PTR_LEN);
/* Log the preceding fields. */ ASSERT_ZERO(data, src - rec); memcpy(data, rec, ulint(src - rec)); data += src - rec;
/* Store trx_id and roll_ptr. */ memcpy(storage - (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN) * (heap_no - 1), src, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
src += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN;
/* Log the last bytes of the record. */ len = rec_offs_data_size(offsets) - ulint(src - rec);
ASSERT_ZERO(data, len); memcpy(data, src, len); data += len; } } else { /* Leaf page of a secondary index:
no externally stored columns */ ut_ad(!rec_offs_any_extern(offsets));
/* Log the entire record. */ len = rec_offs_data_size(offsets);
ASSERT_ZERO(data, len); memcpy(data, rec, len); data += len; } } else { /* This is a node pointer page. */ ulint len;
/* Non-leaf nodes should not have any externally
stored columns. */ ut_ad(!rec_offs_any_extern(offsets));
/* Copy the data bytes, except node_ptr. */ len = rec_offs_data_size(offsets) - REC_NODE_PTR_SIZE; ut_ad(data + len < storage - REC_NODE_PTR_SIZE * (page_dir_get_n_heap(page) - PAGE_HEAP_NO_USER_LOW)); ASSERT_ZERO(data, len); memcpy(data, rec, len); data += len;
/* Copy the node pointer to the uncompressed area. */ memcpy(storage - REC_NODE_PTR_SIZE * (heap_no - 1), rec + len, REC_NODE_PTR_SIZE); }
ut_a(!*data); ut_ad((ulint) (data - page_zip->data) < page_zip_get_size(page_zip)); page_zip->m_end = unsigned(data - page_zip->data); page_zip->m_nonempty = TRUE;
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page_align(rec), index));#endif /* UNIV_ZIP_DEBUG */
}
/***********************************************************//**
Parses a log record of writing a BLOB pointer of a record.@return end of log record or NULL */const byte*page_zip_parse_write_blob_ptr(/*==========================*/ const byte* ptr, /*!< in: redo log buffer */ const byte* end_ptr,/*!< in: redo log buffer end */ page_t* page, /*!< in/out: uncompressed page */ page_zip_des_t* page_zip)/*!< in/out: compressed page */{ ulint offset; ulint z_offset;
ut_ad(ptr != NULL); ut_ad(end_ptr != NULL); ut_ad(!page == !page_zip);
if (UNIV_UNLIKELY (end_ptr < ptr + (2 + 2 + BTR_EXTERN_FIELD_REF_SIZE))) {
return(NULL); }
offset = mach_read_from_2(ptr); z_offset = mach_read_from_2(ptr + 2);
if (offset < PAGE_ZIP_START || offset >= srv_page_size || z_offset >= srv_page_size) {corrupt: recv_sys.found_corrupt_log = TRUE;
return(NULL); }
if (page) {
if (!page_zip || !page_is_leaf(page)) {
goto corrupt; }
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
memcpy(page + offset, ptr + 4, BTR_EXTERN_FIELD_REF_SIZE); memcpy(page_zip->data + z_offset, ptr + 4, BTR_EXTERN_FIELD_REF_SIZE);
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
}
return(ptr + (2 + 2 + BTR_EXTERN_FIELD_REF_SIZE));}
/**********************************************************************//**
Write a BLOB pointer of a record on the leaf page of a clustered index.The information must already have been updated on the uncompressed page. */voidpage_zip_write_blob_ptr(/*====================*/ buf_block_t* block, /*!< in/out: ROW_FORMAT=COMPRESSED page */ const byte* rec, /*!< in/out: record whose data is being
written */ dict_index_t* index, /*!< in: index of the page */ const offset_t* offsets,/*!< in: rec_get_offsets(rec, index) */ ulint n, /*!< in: column index */ mtr_t* mtr) /*!< in/out: mini-transaction */{ const byte* field; byte* externs; const page_t* const page = block->frame; page_zip_des_t* const page_zip = &block->page.zip; ulint blob_no; ulint len;
ut_ad(page_align(rec) == page); ut_ad(index != NULL); ut_ad(offsets != NULL); ut_ad(page_simple_validate_new((page_t*) page)); ut_ad(page_zip_simple_validate(page_zip)); ut_ad(page_zip_get_size(page_zip) > PAGE_DATA + page_zip_dir_size(page_zip)); ut_ad(rec_offs_comp(offsets)); ut_ad(rec_offs_validate(rec, NULL, offsets)); ut_ad(rec_offs_any_extern(offsets)); ut_ad(rec_offs_nth_extern(offsets, n));
ut_ad(page_zip->m_start >= PAGE_DATA); ut_ad(page_zip_header_cmp(page_zip, page));
ut_ad(page_is_leaf(page)); ut_ad(dict_index_is_clust(index));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
blob_no = page_zip_get_n_prev_extern(page_zip, rec, index) + rec_get_n_extern_new(rec, index, n); ut_a(blob_no < page_zip->n_blobs);
externs = page_zip->data + page_zip_get_size(page_zip) - (page_dir_get_n_heap(page) - PAGE_HEAP_NO_USER_LOW) * PAGE_ZIP_CLUST_LEAF_SLOT_SIZE;
field = rec_get_nth_field(rec, offsets, n, &len);
externs -= (blob_no + 1) * BTR_EXTERN_FIELD_REF_SIZE; field += len - BTR_EXTERN_FIELD_REF_SIZE;
memcpy(externs, field, BTR_EXTERN_FIELD_REF_SIZE);
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, index));#endif /* UNIV_ZIP_DEBUG */
if (byte* log_ptr = mlog_open(mtr, 11 + 2 + 2 + FIELD_REF_SIZE)) { log_ptr = mlog_write_initial_log_record_low( MLOG_ZIP_WRITE_BLOB_PTR, block->page.id.space(), block->page.id.page_no(), log_ptr, mtr); mach_write_to_2(log_ptr, page_offset(field)); log_ptr += 2; mach_write_to_2(log_ptr, ulint(externs - page_zip->data)); log_ptr += 2; memcpy(log_ptr, externs, BTR_EXTERN_FIELD_REF_SIZE); log_ptr += BTR_EXTERN_FIELD_REF_SIZE; mlog_close(mtr, log_ptr); }}
/***********************************************************//**
Parses a log record of writing the node pointer of a record.@return end of log record or NULL */const byte*page_zip_parse_write_node_ptr(/*==========================*/ const byte* ptr, /*!< in: redo log buffer */ const byte* end_ptr,/*!< in: redo log buffer end */ page_t* page, /*!< in/out: uncompressed page */ page_zip_des_t* page_zip)/*!< in/out: compressed page */{ ulint offset; ulint z_offset;
ut_ad(ptr != NULL); ut_ad(end_ptr!= NULL); ut_ad(!page == !page_zip);
if (UNIV_UNLIKELY(end_ptr < ptr + (2 + 2 + REC_NODE_PTR_SIZE))) {
return(NULL); }
offset = mach_read_from_2(ptr); z_offset = mach_read_from_2(ptr + 2);
if (offset < PAGE_ZIP_START || offset >= srv_page_size || z_offset >= srv_page_size) {corrupt: recv_sys.found_corrupt_log = TRUE;
return(NULL); }
if (page) { byte* storage_end; byte* field; byte* storage; ulint heap_no;
if (!page_zip || page_is_leaf(page)) {
goto corrupt; }
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
field = page + offset; storage = page_zip->data + z_offset;
storage_end = page_zip_dir_start(page_zip);
heap_no = 1 + ulint(storage_end - storage) / REC_NODE_PTR_SIZE;
if (UNIV_UNLIKELY((storage_end - storage) % REC_NODE_PTR_SIZE) || UNIV_UNLIKELY(heap_no < PAGE_HEAP_NO_USER_LOW) || UNIV_UNLIKELY(heap_no >= page_dir_get_n_heap(page))) {
goto corrupt; }
memcpy(field, ptr + 4, REC_NODE_PTR_SIZE); memcpy(storage, ptr + 4, REC_NODE_PTR_SIZE);
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
}
return(ptr + (2 + 2 + REC_NODE_PTR_SIZE));}
/**********************************************************************//**
Write the node pointer of a record on a non-leaf compressed page. */voidpage_zip_write_node_ptr(/*====================*/ buf_block_t* block, /*!< in/out: compressed page */ byte* rec, /*!< in/out: record */ ulint size, /*!< in: data size of rec */ ulint ptr, /*!< in: node pointer */ mtr_t* mtr) /*!< in/out: mini-transaction */{ byte* field; byte* storage; page_zip_des_t* const page_zip = &block->page.zip;
ut_d(const page_t* const page = block->frame); ut_ad(page_simple_validate_new(page)); ut_ad(page_zip_simple_validate(page_zip)); ut_ad(page_zip_get_size(page_zip) > PAGE_DATA + page_zip_dir_size(page_zip)); ut_ad(page_rec_is_comp(rec));
ut_ad(page_zip->m_start >= PAGE_DATA); ut_ad(page_zip_header_cmp(page_zip, page));
ut_ad(!page_is_leaf(page));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(rec, size);
storage = page_zip_dir_start(page_zip) - (rec_get_heap_no_new(rec) - 1) * REC_NODE_PTR_SIZE; field = rec + size - REC_NODE_PTR_SIZE;
#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
ut_a(!memcmp(storage, field, REC_NODE_PTR_SIZE));#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
compile_time_assert(REC_NODE_PTR_SIZE == 4); mach_write_to_4(field, ptr); memcpy(storage, field, REC_NODE_PTR_SIZE);
if (byte* log_ptr = mlog_open(mtr, 11 + 2 + 2 + REC_NODE_PTR_SIZE)) { log_ptr = mlog_write_initial_log_record_low( MLOG_ZIP_WRITE_NODE_PTR, block->page.id.space(), block->page.id.page_no(), log_ptr, mtr); mach_write_to_2(log_ptr, page_offset(field)); log_ptr += 2; mach_write_to_2(log_ptr, ulint(storage - page_zip->data)); log_ptr += 2; memcpy(log_ptr, field, REC_NODE_PTR_SIZE); log_ptr += REC_NODE_PTR_SIZE; mlog_close(mtr, log_ptr); }}
/** Write the DB_TRX_ID,DB_ROLL_PTR into a clustered index leaf page record.
@param[in,out] page_zip compressed page@param[in,out] rec record@param[in] offsets rec_get_offsets(rec, index)@param[in] trx_id_field field number of DB_TRX_ID (number of PK fields)@param[in] trx_id DB_TRX_ID value (transaction identifier)@param[in] roll_ptr DB_ROLL_PTR value (undo log pointer)@param[in,out] mtr mini-transaction, or NULL to skip logging */voidpage_zip_write_trx_id_and_roll_ptr( page_zip_des_t* page_zip, byte* rec, const offset_t* offsets, ulint trx_id_col, trx_id_t trx_id, roll_ptr_t roll_ptr, mtr_t* mtr){ byte* field; byte* storage;#ifdef UNIV_DEBUG
page_t* page = page_align(rec);#endif /* UNIV_DEBUG */
ulint len;
ut_ad(page_simple_validate_new(page)); ut_ad(page_zip_simple_validate(page_zip)); ut_ad(page_zip_get_size(page_zip) > PAGE_DATA + page_zip_dir_size(page_zip)); ut_ad(rec_offs_validate(rec, NULL, offsets)); ut_ad(rec_offs_comp(offsets));
ut_ad(page_zip->m_start >= PAGE_DATA); ut_ad(page_zip_header_cmp(page_zip, page));
ut_ad(page_is_leaf(page));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
storage = page_zip_dir_start(page_zip) - (rec_get_heap_no_new(rec) - 1) * (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
compile_time_assert(DATA_TRX_ID + 1 == DATA_ROLL_PTR); field = rec_get_nth_field(rec, offsets, trx_id_col, &len); ut_ad(len == DATA_TRX_ID_LEN); ut_ad(field + DATA_TRX_ID_LEN == rec_get_nth_field(rec, offsets, trx_id_col + 1, &len)); ut_ad(len == DATA_ROLL_PTR_LEN);#if defined UNIV_DEBUG || defined UNIV_ZIP_DEBUG
ut_a(!memcmp(storage, field, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN));#endif /* UNIV_DEBUG || UNIV_ZIP_DEBUG */
compile_time_assert(DATA_TRX_ID_LEN == 6); mach_write_to_6(field, trx_id); compile_time_assert(DATA_ROLL_PTR_LEN == 7); mach_write_to_7(field + DATA_TRX_ID_LEN, roll_ptr); memcpy(storage, field, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets)); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
if (mtr) { byte* log_ptr = mlog_open( mtr, 11 + 2 + 2 + DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); if (UNIV_UNLIKELY(!log_ptr)) { return; }
log_ptr = mlog_write_initial_log_record_fast( (byte*) field, MLOG_ZIP_WRITE_TRX_ID, log_ptr, mtr); mach_write_to_2(log_ptr, page_offset(field)); log_ptr += 2; mach_write_to_2(log_ptr, ulint(storage - page_zip->data)); log_ptr += 2; memcpy(log_ptr, field, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); log_ptr += DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN; mlog_close(mtr, log_ptr); }}
/** Parse a MLOG_ZIP_WRITE_TRX_ID record.
@param[in] ptr redo log buffer@param[in] end_ptr end of redo log buffer@param[in,out] page uncompressed page@param[in,out] page_zip compressed page@return end of log record@retval NULL if the log record is incomplete */const byte*page_zip_parse_write_trx_id( const byte* ptr, const byte* end_ptr, page_t* page, page_zip_des_t* page_zip){ const byte* const end = 2 + 2 + DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN + ptr;
if (UNIV_UNLIKELY(end_ptr < end)) { return(NULL); }
uint offset = mach_read_from_2(ptr); uint z_offset = mach_read_from_2(ptr + 2);
if (offset < PAGE_ZIP_START || offset >= srv_page_size || z_offset >= srv_page_size) {corrupt: recv_sys.found_corrupt_log = TRUE;
return(NULL); }
if (page) { if (!page_zip || !page_is_leaf(page)) { goto corrupt; }
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
byte* field = page + offset; byte* storage = page_zip->data + z_offset;
if (storage >= page_zip_dir_start(page_zip)) { goto corrupt; }
memcpy(field, ptr + 4, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); memcpy(storage, ptr + 4, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
}
return end;}
/**********************************************************************//**
Clear an area on the uncompressed and compressed page.Do not clear the data payload, as that would grow the modification log. */staticvoidpage_zip_clear_rec(/*===============*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ byte* rec, /*!< in: record to clear */ const dict_index_t* index, /*!< in: index of rec */ const offset_t* offsets) /*!< in: rec_get_offsets(rec, index) */{ ulint heap_no; page_t* page = page_align(rec); byte* storage; byte* field; ulint len; /* page_zip_validate() would fail here if a record
containing externally stored columns is being deleted. */ ut_ad(rec_offs_validate(rec, index, offsets)); ut_ad(!page_zip_dir_find(page_zip, page_offset(rec))); ut_ad(page_zip_dir_find_free(page_zip, page_offset(rec))); ut_ad(page_zip_header_cmp(page_zip, page));
heap_no = rec_get_heap_no_new(rec); ut_ad(heap_no >= PAGE_HEAP_NO_USER_LOW);
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
if (!page_is_leaf(page)) { /* Clear node_ptr. On the compressed page,
there is an array of node_ptr immediately before the dense page directory, at the very end of the page. */ storage = page_zip_dir_start(page_zip); ut_ad(dict_index_get_n_unique_in_tree_nonleaf(index) == rec_offs_n_fields(offsets) - 1); field = rec_get_nth_field(rec, offsets, rec_offs_n_fields(offsets) - 1, &len); ut_ad(len == REC_NODE_PTR_SIZE);
ut_ad(!rec_offs_any_extern(offsets)); memset(field, 0, REC_NODE_PTR_SIZE); memset(storage - (heap_no - 1) * REC_NODE_PTR_SIZE, 0, REC_NODE_PTR_SIZE); } else if (dict_index_is_clust(index)) { /* Clear trx_id and roll_ptr. On the compressed page,
there is an array of these fields immediately before the dense page directory, at the very end of the page. */ const ulint trx_id_pos = dict_col_get_clust_pos( dict_table_get_sys_col( index->table, DATA_TRX_ID), index); storage = page_zip_dir_start(page_zip); field = rec_get_nth_field(rec, offsets, trx_id_pos, &len); ut_ad(len == DATA_TRX_ID_LEN);
memset(field, 0, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); memset(storage - (heap_no - 1) * (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN), 0, DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN);
if (rec_offs_any_extern(offsets)) { ulint i;
for (i = rec_offs_n_fields(offsets); i--; ) { /* Clear all BLOB pointers in order to make
page_zip_validate() pass. */ if (rec_offs_nth_extern(offsets, i)) { field = rec_get_nth_field( rec, offsets, i, &len); ut_ad(len == BTR_EXTERN_FIELD_REF_SIZE); memset(field + len - BTR_EXTERN_FIELD_REF_SIZE, 0, BTR_EXTERN_FIELD_REF_SIZE); } } } } else { ut_ad(!rec_offs_any_extern(offsets)); }}
/**********************************************************************//**
Write the "deleted" flag of a record on a compressed page. The flag mustalready have been written on the uncompressed page. */voidpage_zip_rec_set_deleted(/*=====================*/ page_zip_des_t* page_zip,/*!< in/out: compressed page */ const byte* rec, /*!< in: record on the uncompressed page */ ulint flag) /*!< in: the deleted flag (nonzero=TRUE) */{ byte* slot = page_zip_dir_find(page_zip, page_offset(rec)); ut_a(slot); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); if (flag) { *slot |= (PAGE_ZIP_DIR_SLOT_DEL >> 8); } else { *slot &= ~(PAGE_ZIP_DIR_SLOT_DEL >> 8); }#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page_align(rec), NULL));#endif /* UNIV_ZIP_DEBUG */
}
/**********************************************************************//**
Write the "owned" flag of a record on a compressed page. The n_owned fieldmust already have been written on the uncompressed page. */voidpage_zip_rec_set_owned(/*===================*/ page_zip_des_t* page_zip,/*!< in/out: compressed page */ const byte* rec, /*!< in: record on the uncompressed page */ ulint flag) /*!< in: the owned flag (nonzero=TRUE) */{ byte* slot = page_zip_dir_find(page_zip, page_offset(rec)); ut_a(slot); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); if (flag) { *slot |= (PAGE_ZIP_DIR_SLOT_OWNED >> 8); } else { *slot &= ~(PAGE_ZIP_DIR_SLOT_OWNED >> 8); }}
/**********************************************************************//**
Insert a record to the dense page directory. */voidpage_zip_dir_insert(/*================*/ page_cur_t* cursor, /*!< in/out: page cursor */ const byte* free_rec,/*!< in: record from which rec was
allocated, or NULL */ byte* rec) /*!< in: record to insert */{ ut_ad(page_align(cursor->rec) == cursor->block->frame); ut_ad(page_align(rec) == cursor->block->frame); page_zip_des_t *const page_zip= &cursor->block->page.zip;
ulint n_dense; byte* slot_rec; byte* slot_free;
ut_ad(cursor->rec != rec); ut_ad(page_rec_get_next_const(cursor->rec) == rec); ut_ad(page_zip_simple_validate(page_zip));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
if (page_rec_is_infimum(cursor->rec)) { /* Use the first slot. */ slot_rec = page_zip->data + page_zip_get_size(page_zip); } else { byte* end = page_zip->data + page_zip_get_size(page_zip); byte* start = end - page_zip_dir_user_size(page_zip);
if (UNIV_LIKELY(!free_rec)) { /* PAGE_N_RECS was already incremented
in page_cur_insert_rec_zip(), but the dense directory slot at that position contains garbage. Skip it. */ start += PAGE_ZIP_DIR_SLOT_SIZE; }
slot_rec = page_zip_dir_find_low(start, end, page_offset(cursor->rec)); ut_a(slot_rec); }
/* Read the old n_dense (n_heap may have been incremented). */ n_dense = page_dir_get_n_heap(page_zip->data) - (PAGE_HEAP_NO_USER_LOW + 1U);
if (UNIV_LIKELY_NULL(free_rec)) { /* The record was allocated from the free list.
Shift the dense directory only up to that slot. Note that in this case, n_dense is actually off by one, because page_cur_insert_rec_zip() did not increment n_heap. */ ut_ad(rec_get_heap_no_new(rec) < n_dense + 1 + PAGE_HEAP_NO_USER_LOW); ut_ad(rec >= free_rec); slot_free = page_zip_dir_find(page_zip, page_offset(free_rec)); ut_ad(slot_free); slot_free += PAGE_ZIP_DIR_SLOT_SIZE; } else { /* The record was allocated from the heap.
Shift the entire dense directory. */ ut_ad(rec_get_heap_no_new(rec) == n_dense + PAGE_HEAP_NO_USER_LOW);
/* Shift to the end of the dense page directory. */ slot_free = page_zip->data + page_zip_get_size(page_zip) - PAGE_ZIP_DIR_SLOT_SIZE * n_dense; }
/* Shift the dense directory to allocate place for rec. */ memmove_aligned<2>(slot_free - PAGE_ZIP_DIR_SLOT_SIZE, slot_free, ulint(slot_rec - slot_free));
/* Write the entry for the inserted record.
The "owned" and "deleted" flags must be zero. */ mach_write_to_2(slot_rec - PAGE_ZIP_DIR_SLOT_SIZE, page_offset(rec));}
/**********************************************************************//**
Shift the dense page directory and the array of BLOB pointerswhen a record is deleted. */voidpage_zip_dir_delete(/*================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ byte* rec, /*!< in: deleted record */ const dict_index_t* index, /*!< in: index of rec */ const offset_t* offsets, /*!< in: rec_get_offsets(rec) */ const byte* free) /*!< in: previous start of
the free list */{ byte* slot_rec; byte* slot_free; ulint n_ext; page_t* page = page_align(rec);
ut_ad(rec_offs_validate(rec, index, offsets)); ut_ad(rec_offs_comp(offsets));
UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(rec, rec_offs_data_size(offsets)); UNIV_MEM_ASSERT_RW(rec - rec_offs_extra_size(offsets), rec_offs_extra_size(offsets));
slot_rec = page_zip_dir_find(page_zip, page_offset(rec));
ut_a(slot_rec); uint16_t n_recs = page_get_n_recs(page); ut_ad(n_recs); ut_ad(n_recs > 1 || page_get_page_no(page) == index->page); /* This could not be done before page_zip_dir_find(). */ page_header_set_field(page, page_zip, PAGE_N_RECS, n_recs - 1);
if (UNIV_UNLIKELY(!free)) { /* Make the last slot the start of the free list. */ slot_free = page_zip->data + page_zip_get_size(page_zip) - PAGE_ZIP_DIR_SLOT_SIZE * (page_dir_get_n_heap(page_zip->data) - PAGE_HEAP_NO_USER_LOW); } else { slot_free = page_zip_dir_find_free(page_zip, page_offset(free)); ut_a(slot_free < slot_rec); /* Grow the free list by one slot by moving the start. */ slot_free += PAGE_ZIP_DIR_SLOT_SIZE; }
if (UNIV_LIKELY(slot_rec > slot_free)) { memmove_aligned<2>(slot_free + PAGE_ZIP_DIR_SLOT_SIZE, slot_free, ulint(slot_rec - slot_free)); }
/* Write the entry for the deleted record.
The "owned" and "deleted" flags will be cleared. */ mach_write_to_2(slot_free, page_offset(rec));
if (!page_is_leaf(page) || !dict_index_is_clust(index)) { ut_ad(!rec_offs_any_extern(offsets)); goto skip_blobs; }
n_ext = rec_offs_n_extern(offsets); if (UNIV_UNLIKELY(n_ext != 0)) { /* Shift and zero fill the array of BLOB pointers. */ ulint blob_no; byte* externs; byte* ext_end;
blob_no = page_zip_get_n_prev_extern(page_zip, rec, index); ut_a(blob_no + n_ext <= page_zip->n_blobs);
externs = page_zip->data + page_zip_get_size(page_zip) - (page_dir_get_n_heap(page) - PAGE_HEAP_NO_USER_LOW) * PAGE_ZIP_CLUST_LEAF_SLOT_SIZE;
ext_end = externs - page_zip->n_blobs * BTR_EXTERN_FIELD_REF_SIZE; externs -= blob_no * BTR_EXTERN_FIELD_REF_SIZE;
page_zip->n_blobs -= static_cast<unsigned>(n_ext); /* Shift and zero fill the array. */ memmove(ext_end + n_ext * BTR_EXTERN_FIELD_REF_SIZE, ext_end, ulint(page_zip->n_blobs - blob_no) * BTR_EXTERN_FIELD_REF_SIZE); memset(ext_end, 0, n_ext * BTR_EXTERN_FIELD_REF_SIZE); }
skip_blobs: /* The compression algorithm expects info_bits and n_owned
to be 0 for deleted records. */ rec[-REC_N_NEW_EXTRA_BYTES] = 0; /* info_bits and n_owned */
page_zip_clear_rec(page_zip, rec, index, offsets);}
/**********************************************************************//**
Add a slot to the dense page directory. */voidpage_zip_dir_add_slot(/*==================*/ page_zip_des_t* page_zip, /*!< in/out: compressed page */ ulint is_clustered) /*!< in: nonzero for clustered index,
zero for others */{ ulint n_dense; byte* dir; byte* stored;
ut_ad(page_is_comp(page_zip->data)); UNIV_MEM_ASSERT_RW(page_zip->data, page_zip_get_size(page_zip));
/* Read the old n_dense (n_heap has already been incremented). */ n_dense = page_dir_get_n_heap(page_zip->data) - (PAGE_HEAP_NO_USER_LOW + 1U);
dir = page_zip->data + page_zip_get_size(page_zip) - PAGE_ZIP_DIR_SLOT_SIZE * n_dense;
if (!page_is_leaf(page_zip->data)) { ut_ad(!page_zip->n_blobs); stored = dir - n_dense * REC_NODE_PTR_SIZE; } else if (is_clustered) { /* Move the BLOB pointer array backwards to make space for the
roll_ptr and trx_id columns and the dense directory slot. */ byte* externs;
stored = dir - n_dense * (DATA_TRX_ID_LEN + DATA_ROLL_PTR_LEN); externs = stored - page_zip->n_blobs * BTR_EXTERN_FIELD_REF_SIZE; ASSERT_ZERO(externs - PAGE_ZIP_CLUST_LEAF_SLOT_SIZE, PAGE_ZIP_CLUST_LEAF_SLOT_SIZE); memmove(externs - PAGE_ZIP_CLUST_LEAF_SLOT_SIZE, externs, ulint(stored - externs)); } else { stored = dir - page_zip->n_blobs * BTR_EXTERN_FIELD_REF_SIZE; ASSERT_ZERO(stored - PAGE_ZIP_DIR_SLOT_SIZE, static_cast<size_t>(PAGE_ZIP_DIR_SLOT_SIZE)); }
/* Move the uncompressed area backwards to make space
for one directory slot. */ memmove(stored - PAGE_ZIP_DIR_SLOT_SIZE, stored, ulint(dir - stored));}
/***********************************************************//**
Parses a log record of writing to the header of a page.@return end of log record or NULL */const byte*page_zip_parse_write_header(/*========================*/ const byte* ptr, /*!< in: redo log buffer */ const byte* end_ptr,/*!< in: redo log buffer end */ page_t* page, /*!< in/out: uncompressed page */ page_zip_des_t* page_zip)/*!< in/out: compressed page */{ ulint offset; ulint len;
ut_ad(ptr != NULL); ut_ad(end_ptr!= NULL); ut_ad(!page == !page_zip);
if (UNIV_UNLIKELY(end_ptr < ptr + (1 + 1))) {
return(NULL); }
offset = (ulint) *ptr++; len = (ulint) *ptr++;
if (len == 0 || offset + len >= PAGE_DATA) {corrupt: recv_sys.found_corrupt_log = TRUE;
return(NULL); }
if (end_ptr < ptr + len) {
return(NULL); }
if (page) { if (!page_zip) {
goto corrupt; }#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
memcpy(page + offset, ptr, len); memcpy(page_zip->data + offset, ptr, len);
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, NULL));#endif /* UNIV_ZIP_DEBUG */
}
return(ptr + len);}
/**********************************************************************//**
Write a log record of writing to the uncompressed header portion of a page. */voidpage_zip_write_header_log(/*======================*/ const byte* data, /*!< in: data on the uncompressed page */ ulint length, /*!< in: length of the data */ mtr_t* mtr) /*!< in: mini-transaction */{ byte* log_ptr = mlog_open(mtr, 11 + 1 + 1); ulint offset = page_offset(data);
ut_ad(offset < PAGE_DATA); ut_ad(offset + length < PAGE_DATA); compile_time_assert(PAGE_DATA < 256U); ut_ad(length > 0); ut_ad(length < 256);
/* If no logging is requested, we may return now */ if (UNIV_UNLIKELY(!log_ptr)) {
return; }
log_ptr = mlog_write_initial_log_record_fast( (byte*) data, MLOG_ZIP_WRITE_HEADER, log_ptr, mtr); *log_ptr++ = (byte) offset; *log_ptr++ = (byte) length; mlog_close(mtr, log_ptr);
mlog_catenate_string(mtr, data, length);}
/**********************************************************************//**
Reorganize and compress a page. This is a low-level operation forcompressed pages, to be used when page_zip_compress() fails.On success, a redo log entry MLOG_ZIP_PAGE_COMPRESS will be written.The function btr_page_reorganize() should be preferred whenever possible.IMPORTANT: if page_zip_reorganize() is invoked on a leaf page of anon-clustered index, the caller must update the insert buffer freebits in the same mini-transaction in such a way that the modificationwill be redo-logged.@return TRUE on success, FALSE on failure; page_zip will be leftintact on failure, but page will be overwritten. */iboolpage_zip_reorganize(/*================*/ buf_block_t* block, /*!< in/out: page with compressed page;
on the compressed page, in: size; out: data, n_blobs, m_start, m_end, m_nonempty */ dict_index_t* index, /*!< in: index of the B-tree node */ mtr_t* mtr) /*!< in: mini-transaction */{ buf_pool_t* buf_pool = buf_pool_from_block(block); page_t* page = buf_block_get_frame(block); buf_block_t* temp_block; page_t* temp_page;
ut_ad(mtr_memo_contains(mtr, block, MTR_MEMO_PAGE_X_FIX)); ut_ad(page_is_comp(page)); ut_ad(!dict_index_is_ibuf(index)); ut_ad(!index->table->is_temporary()); /* Note that page_zip_validate(page_zip, page, index) may fail here. */ UNIV_MEM_ASSERT_RW(page, srv_page_size); UNIV_MEM_ASSERT_RW(buf_block_get_page_zip(block)->data, page_zip_get_size(buf_block_get_page_zip(block)));
/* Disable logging */ mtr_log_t log_mode = mtr_set_log_mode(mtr, MTR_LOG_NONE);
temp_block = buf_block_alloc(buf_pool); btr_search_drop_page_hash_index(block); temp_page = temp_block->frame;
/* Copy the old page to temporary space */ memcpy_aligned<UNIV_PAGE_SIZE_MIN>(temp_block->frame, block->frame, srv_page_size);
/* Recreate the page: note that global data on page (possible
segment headers, next page-field, etc.) is preserved intact */
page_create(block, mtr, TRUE, dict_index_is_spatial(index));
/* Copy the records from the temporary space to the recreated page;
do not copy the lock bits yet */
page_copy_rec_list_end_no_locks(block, temp_block, page_get_infimum_rec(temp_page), index, mtr);
/* Copy the PAGE_MAX_TRX_ID or PAGE_ROOT_AUTO_INC. */ memcpy_aligned<8>(page + (PAGE_HEADER + PAGE_MAX_TRX_ID), temp_page + (PAGE_HEADER + PAGE_MAX_TRX_ID), 8); /* PAGE_MAX_TRX_ID must be set on secondary index leaf pages. */ ut_ad(dict_index_is_clust(index) || !page_is_leaf(temp_page) || page_get_max_trx_id(page) != 0); /* PAGE_MAX_TRX_ID must be zero on non-leaf pages other than
clustered index root pages. */ ut_ad(page_get_max_trx_id(page) == 0 || (dict_index_is_clust(index) ? !page_has_siblings(temp_page) : page_is_leaf(temp_page)));
/* Restore logging. */ mtr_set_log_mode(mtr, log_mode);
if (!page_zip_compress(block, index, page_zip_level, mtr)) { buf_block_free(temp_block); return(FALSE); }
lock_move_reorganize_page(block, temp_block);
buf_block_free(temp_block); return(TRUE);}
/**********************************************************************//**
Copy the records of a page byte for byte. Do not copy the page headeror trailer, except those B-tree header fields that are directlyrelated to the storage of records. Also copy PAGE_MAX_TRX_ID.NOTE: The caller must update the lock table and the adaptive hash index. */voidpage_zip_copy_recs( buf_block_t* block, /*!< in/out: buffer block */ const page_zip_des_t* src_zip, /*!< in: compressed page */ const page_t* src, /*!< in: page */ dict_index_t* index, /*!< in: index of the B-tree */ mtr_t* mtr) /*!< in: mini-transaction */{ page_t* page = block->frame; page_zip_des_t* page_zip = &block->page.zip;
ut_ad(mtr_memo_contains_page(mtr, page, MTR_MEMO_PAGE_X_FIX)); ut_ad(mtr_memo_contains_page(mtr, src, MTR_MEMO_PAGE_X_FIX)); ut_ad(!dict_index_is_ibuf(index)); ut_ad(!index->table->is_temporary());#ifdef UNIV_ZIP_DEBUG
/* The B-tree operations that call this function may set
FIL_PAGE_PREV or PAGE_LEVEL, causing a temporary min_rec_flag mismatch. A strict page_zip_validate() will be executed later during the B-tree operations. */ ut_a(page_zip_validate_low(src_zip, src, index, TRUE));#endif /* UNIV_ZIP_DEBUG */
ut_a(page_zip_get_size(page_zip) == page_zip_get_size(src_zip)); if (UNIV_UNLIKELY(src_zip->n_blobs)) { ut_a(page_is_leaf(src)); ut_a(dict_index_is_clust(index)); }
UNIV_MEM_ASSERT_W(page, srv_page_size); UNIV_MEM_ASSERT_W(page_zip->data, page_zip_get_size(page_zip)); UNIV_MEM_ASSERT_RW(src, srv_page_size); UNIV_MEM_ASSERT_RW(src_zip->data, page_zip_get_size(page_zip));
/* Copy those B-tree page header fields that are related to
the records stored in the page. Also copy the field PAGE_MAX_TRX_ID. Skip the rest of the page header and trailer. On the compressed page, there is no trailer. */ compile_time_assert(PAGE_MAX_TRX_ID + 8 == PAGE_HEADER_PRIV_END); memcpy_aligned<2>(PAGE_HEADER + page, PAGE_HEADER + src, PAGE_HEADER_PRIV_END); memcpy_aligned<2>(PAGE_DATA + page, PAGE_DATA + src, srv_page_size - (PAGE_DATA + FIL_PAGE_DATA_END)); memcpy_aligned<2>(PAGE_HEADER + page_zip->data, PAGE_HEADER + src_zip->data, PAGE_HEADER_PRIV_END); memcpy_aligned<2>(PAGE_DATA + page_zip->data, PAGE_DATA + src_zip->data, page_zip_get_size(page_zip) - PAGE_DATA);
if (dict_index_is_clust(index)) { /* Reset the PAGE_ROOT_AUTO_INC field when copying
from a root page. */ memset_aligned<8>(PAGE_HEADER + PAGE_ROOT_AUTO_INC + page, 0, 8); memset_aligned<8>(PAGE_HEADER + PAGE_ROOT_AUTO_INC + page_zip->data, 0, 8); } else { /* The PAGE_MAX_TRX_ID must be nonzero on leaf pages
of secondary indexes, and 0 on others. */ ut_ad(!page_is_leaf(src) == !page_get_max_trx_id(src)); }
/* Copy all fields of src_zip to page_zip, except the pointer
to the compressed data page. */ { page_zip_t* data = page_zip->data; memcpy(page_zip, src_zip, sizeof *page_zip); page_zip->data = data; } ut_ad(page_zip_get_trailer_len(page_zip, dict_index_is_clust(index)) + page_zip->m_end < page_zip_get_size(page_zip));
if (!page_is_leaf(src) && UNIV_UNLIKELY(!page_has_prev(src)) && UNIV_LIKELY(page_has_prev(page))) { /* Clear the REC_INFO_MIN_REC_FLAG of the first user record. */ ulint offs = rec_get_next_offs(page + PAGE_NEW_INFIMUM, TRUE); if (UNIV_LIKELY(offs != PAGE_NEW_SUPREMUM)) { rec_t* rec = page + offs; ut_a(rec[-REC_N_NEW_EXTRA_BYTES] & REC_INFO_MIN_REC_FLAG); rec[-REC_N_NEW_EXTRA_BYTES] &= ~ REC_INFO_MIN_REC_FLAG; } }
#ifdef UNIV_ZIP_DEBUG
ut_a(page_zip_validate(page_zip, page, index));#endif /* UNIV_ZIP_DEBUG */
page_zip_compress_write_log(block, index, mtr);}
/** Parse and optionally apply MLOG_ZIP_PAGE_COMPRESS.
@param[in] ptr log record@param[in] end_ptr end of log@param[in,out] block ROW_FORMAT=COMPRESSED block, or NULL for parsing only@return end of log record@retval NULL if the log record is incomplete */const byte* page_zip_parse_compress(const byte* ptr, const byte* end_ptr, buf_block_t* block){ ulint size; ulint trailer_size;
ut_ad(ptr != NULL); ut_ad(end_ptr!= NULL);
if (UNIV_UNLIKELY(ptr + (2 + 2) > end_ptr)) {
return(NULL); }
size = mach_read_from_2(ptr); ptr += 2; trailer_size = mach_read_from_2(ptr); ptr += 2;
if (UNIV_UNLIKELY(ptr + 8 + size + trailer_size > end_ptr)) {
return(NULL); }
if (block) { ut_ad(buf_block_get_state(block) == BUF_BLOCK_FILE_PAGE); page_zip_des_t* page_zip = buf_block_get_page_zip(block); if (!page_zip || page_zip_get_size(page_zip) < size || block->page.id.page_no() < 3) {corrupt: recv_sys.found_corrupt_log = TRUE;
return(NULL); }
memset(page_zip->data, 0, page_zip_get_size(page_zip)); mach_write_to_4(FIL_PAGE_OFFSET + page_zip->data, block->page.id.page_no()); mach_write_to_4(FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID + page_zip->data, block->page.id.space()); memcpy(page_zip->data + FIL_PAGE_PREV, ptr, 4); memcpy(page_zip->data + FIL_PAGE_NEXT, ptr + 4, 4); memcpy(page_zip->data + FIL_PAGE_TYPE, ptr + 8, size); memset(page_zip->data + FIL_PAGE_TYPE + size, 0, page_zip_get_size(page_zip) - trailer_size - (FIL_PAGE_TYPE + size)); memcpy(page_zip->data + page_zip_get_size(page_zip) - trailer_size, ptr + 8 + size, trailer_size);
if (UNIV_UNLIKELY(!page_zip_decompress(page_zip, block->frame, TRUE))) {
goto corrupt; } }
return(const_cast<byte*>(ptr) + 8 + size + trailer_size);}#endif /* !UNIV_INNOCHECKSUM */
/** Calculate the compressed page checksum.
@param[in] data compressed page@param[in] size size of compressed page@param[in] algo algorithm to use@return page checksum */uint32_tpage_zip_calc_checksum( const void* data, ulint size, srv_checksum_algorithm_t algo){ uLong adler; const Bytef* s = static_cast<const byte*>(data);
/* Exclude FIL_PAGE_SPACE_OR_CHKSUM, FIL_PAGE_LSN,
and FIL_PAGE_FILE_FLUSH_LSN from the checksum. */
switch (algo) { case SRV_CHECKSUM_ALGORITHM_FULL_CRC32: case SRV_CHECKSUM_ALGORITHM_STRICT_FULL_CRC32: case SRV_CHECKSUM_ALGORITHM_CRC32: case SRV_CHECKSUM_ALGORITHM_STRICT_CRC32: ut_ad(size > FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID); return ut_crc32(s + FIL_PAGE_OFFSET, FIL_PAGE_LSN - FIL_PAGE_OFFSET) ^ ut_crc32(s + FIL_PAGE_TYPE, 2) ^ ut_crc32(s + FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID, size - FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID); case SRV_CHECKSUM_ALGORITHM_INNODB: case SRV_CHECKSUM_ALGORITHM_STRICT_INNODB: ut_ad(size > FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID);
adler = adler32(0L, s + FIL_PAGE_OFFSET, FIL_PAGE_LSN - FIL_PAGE_OFFSET); adler = adler32(adler, s + FIL_PAGE_TYPE, 2); adler = adler32( adler, s + FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID, static_cast<uInt>(size) - FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID);
return(uint32_t(adler)); case SRV_CHECKSUM_ALGORITHM_NONE: case SRV_CHECKSUM_ALGORITHM_STRICT_NONE: return(BUF_NO_CHECKSUM_MAGIC); /* no default so the compiler will emit a warning if new enum
is added and not handled here */ }
ut_error; return(0);}
/** Verify a compressed page's checksum.
@param[in] data compressed page@param[in] size size of compressed page@return whether the stored checksum is valid according to the value ofinnodb_checksum_algorithm */bool page_zip_verify_checksum(const void* data, ulint size){ const uint32_t stored = mach_read_from_4( static_cast<const byte*>(data) + FIL_PAGE_SPACE_OR_CHKSUM);
compile_time_assert(!(FIL_PAGE_LSN % 8));
/* Check if page is empty */ if (stored == 0 && *reinterpret_cast<const ib_uint64_t*>(static_cast<const char*>( data) + FIL_PAGE_LSN) == 0) { /* make sure that the page is really empty */#ifdef UNIV_INNOCHECKSUM
ulint i; for (i = 0; i < size; i++) { if (*((const char*) data + i) != 0) break; } if (i >= size) { if (log_file) { fprintf(log_file, "Page::%llu is empty and" " uncorrupted\n", cur_page_num); }
return(TRUE); }#else
for (ulint i = 0; i < size; i++) { if (*((const char*) data + i) != 0) { return(FALSE); } } /* Empty page */ return(TRUE);#endif /* UNIV_INNOCHECKSUM */
}
const srv_checksum_algorithm_t curr_algo = static_cast<srv_checksum_algorithm_t>(srv_checksum_algorithm);
if (curr_algo == SRV_CHECKSUM_ALGORITHM_NONE) { return(TRUE); }
uint32_t calc = page_zip_calc_checksum(data, size, curr_algo);
#ifdef UNIV_INNOCHECKSUM
if (log_file) { fprintf(log_file, "page::%llu;" " %s checksum: calculated = %u;" " recorded = %u\n", cur_page_num, buf_checksum_algorithm_name( static_cast<srv_checksum_algorithm_t>( srv_checksum_algorithm)), calc, stored); }
if (!strict_verify) {
const uint32_t crc32 = page_zip_calc_checksum( data, size, SRV_CHECKSUM_ALGORITHM_CRC32);
if (log_file) { fprintf(log_file, "page::%llu: crc32 checksum:" " calculated = %u; recorded = %u\n", cur_page_num, crc32, stored); fprintf(log_file, "page::%llu: none checksum:" " calculated = %lu; recorded = %u\n", cur_page_num, BUF_NO_CHECKSUM_MAGIC, stored); } }#endif /* UNIV_INNOCHECKSUM */
if (stored == calc) { return(TRUE); }
switch (curr_algo) { case SRV_CHECKSUM_ALGORITHM_STRICT_FULL_CRC32: case SRV_CHECKSUM_ALGORITHM_STRICT_CRC32: case SRV_CHECKSUM_ALGORITHM_STRICT_INNODB: case SRV_CHECKSUM_ALGORITHM_STRICT_NONE: return FALSE; case SRV_CHECKSUM_ALGORITHM_FULL_CRC32: case SRV_CHECKSUM_ALGORITHM_CRC32: if (stored == BUF_NO_CHECKSUM_MAGIC) { return(TRUE); }
return stored == page_zip_calc_checksum( data, size, SRV_CHECKSUM_ALGORITHM_INNODB); case SRV_CHECKSUM_ALGORITHM_INNODB: if (stored == BUF_NO_CHECKSUM_MAGIC) { return TRUE; }
return stored == page_zip_calc_checksum( data, size, SRV_CHECKSUM_ALGORITHM_CRC32); case SRV_CHECKSUM_ALGORITHM_NONE: return TRUE; }
return FALSE;}
|
