mariadb

Commit Graph

Author	SHA1	Message	Date
Monty	1356b77011	Simple optimization: Remove JOIN::set_group_rpa as it is not needed	4 years ago
Monty	f887fb1243	Added 'records_out' and join_type to POSITION records_out is the numbers of rows expected to be accepted from a table. records_read is in contrast the number of rows that the optimizer excepts to read from the engine. This patch causes not plan changes. The differences in test results comes from renaming "records" to "records_read" and printing of record_out in the optimizer trace. Other things: - Renamed table_cond_selectivity() to table_after_join_selectivity() to make the purpose of the function more clear.	4 years ago
Monty	3f3c51ec3e	Update row and key fetch cost models to take into account data copy costs Before this patch, when calculating the cost of fetching and using a row/key from the engine, we took into account the cost of finding a row or key from the engine, but did not consistently take into account index only accessed, clustered key or covered keys for all access paths. The cost of the WHERE clause (TIME_FOR_COMPARE) was not consistently considered in best_access_path(). TIME_FOR_COMPARE was used in calculation in other places, like greedy_search(), but was in some cases (like scans) done an a different number of rows than was accessed. The cost calculation of row and index scans didn't take into account the number of rows that where accessed, only the number of accepted rows. When using a filter, the cost of index_only_reads and cost of accessing and disregarding 'filtered rows' where not taken into account, which made filters cost less than there actually where. To remedy the above, the following key & row fetch related costs has been added: - The cost of fetching and using a row is now split into different costs: - key + Row fetch cost (as before) but multiplied with the variable 'optimizer_cache_cost' (default to 0.5). This allows the user to tell the optimizer the likehood of finding the key and row in the engine cache. - RECORD_COPY_COST, The cost copying a row from the engine to the sql layer or creating a row from the join_cache to the record buffer. Mostly affects table scan costs. - INDEX_COPY_COST the cost of finding the next key and copying it from the engine to the SQL layer. This is used when we calculate the cost index only reads. It makes index scans more expensive than before if they cover a lot of rows. (main.index_merge_myisam) - INDEX_LOOKUP_COST, the cost of finding the first key in a range. This replaces the old define IDX_LOOKUP_COST, but with a higher cost. - INDEX_NEXT_FIND_COST, the cost of finding the next key (and rowid). when doing a index scan and comparing the rowid to the filter. Before this cost was assumed to be 0. - ROW_LOOKUP_COST, the cost of fetching a row by rowid. All of the above constants/variables are now tuned to be somewhat in proportion of executing complexity to each other. There is tuning need for these in the future, but that can wait until the above are made user variables as that will make tuning much easier. To make the usage of the above easy, there are new (not virtual) cost calclation functions in handler: - ha_read_time(), like read_time(), but take optimizer_cache_cost into account. - ha_read_and_copy_time(), like ha_read_time() but take into account RECORD_COPY_TIME - ha_read_and_compare_time(), like ha_read_and_copy_time() but take TIME_FOR_COMPARE into account. - ha_rndpos_time(). Read row with row id, taking RECORD_COPY_COST into account. This is used with filesort where we don't need to execute the WHERE clause again. - ha_keyread_time(), like keyread_time() but take optimizer_cache_cost into account. - ha_keyread_and_copy_time(), like ha_keyread_time(), but add INDEX_COPY_COST. - ha_key_scan_time(), like key_scan_time() but take optimizer_cache_cost nto account. - ha_key_scan_and_compare_time(), like ha_key_scan_time(), but add INDEX_COPY_COST & TIME_FOR_COMPARE. I also added some setup costs for doing different types of scans and creating temporary tables (on disk and in memory). This encourages the optimizer to not use these for simple 'a few row' lookups if there are adequate key lookup strategies. - TABLE_SCAN_SETUP_COST, cost of starting a table scan. - INDEX_SCAN_SETUP_COST, cost of starting an index scan. - HEAP_TEMPTABLE_CREATE_COST, cost of creating in memory temporary table. - DISK_TEMPTABLE_CREATE_COST, cost of creating an on disk temporary table. When calculating cost of fetching ranges, we had a cost of IDX_LOOKUP_COST (0.125) for doing a key div for a new range. This is now replaced with 'io_cost * INDEX_LOOKUP_COST (1.0) * optimizer_cache_cost', which matches the cost we use for 'ref' and other key lookups. The effect is that the cost is now a bit higher when we have many ranges for a key. Allmost all calculation with TIME_FOR_COMPARE is now done in best_access_path(). 'JOIN::read_time' now includes the full cost for finding the rows in the table. In the result files, many of the changes are now again close to what they where before the "Update cost for hash and cached joins" commit, as that commit didn't fix the filter cost (too complex to do everything in one commit). The above changes showed a lot of a lot of inconsistencies in optimizer cost calculation. The main objective with the other changes was to calculation as similar (and accurate) as possible to make different plans more comparable. Detailed list of changes: - Calculate index_only_cost consistently and correctly for all scan and ref accesses. The row fetch_cost and index_only_cost now takes into account clustered keys, covered keys and index only accesses. - cost_for_index_read now returns both full cost and index_only_cost - Fixed cost calculation of get_sweep_read_cost() to match other similar costs. This is bases on the assumption that data is more often stored on SSD than a hard disk. - Replaced constant 2.0 with new define TABLE_SCAN_SETUP_COST. - Some scan cost estimates did not take into account TIME_FOR_COMPARE. Now all scan costs takes this into account. (main.show_explain) - Added session variable optimizer_cache_hit_ratio (default 50%). By adjusting this on can reduce or increase the cost of index or direct record lookups. The effect of the default is that key lookups is now a bit cheaper than before. See usage of 'optimizer_cache_cost' in handler.h. - JOIN_TAB::scan_time() did not take into account index only scans, which produced a wrong cost when index scan was used. Changed JOIN_TAB:::scan_time() to take into consideration clustered and covered keys. The values are now cached and we only have to call this function once. Other calls are changed to use the cached values. Function renamed to JOIN_TAB::estimate_scan_time(). - Fixed that most index cost calculations are done the same way and more close to 'range' calculations. The cost is now lower than before for small data sets and higher for large data sets as we take into account how many keys are read (main.opt_trace_selectivity, main.limit_rows_examined). - Ensured that index_scan_cost() == range(scan_of_all_rows_in_table_using_one_range) + MULTI_RANGE_READ_INFO_CONST. One effect of this is that if there is choice of doing a full index scan and a range-index scan over almost the whole table then index scan will be preferred (no range-read setup cost). (innodb.innodb, main.show_explain, main.range) - Fixed the EQ_REF and REF takes into account clustered and covered keys. This changes some plans to use covered or clustered indexes as these are much cheaper. (main.subselect_mat_cost, main.state_tables_innodb, main.limit_rows_examined) - Rowid filter setup cost and filter compare cost now takes into account fetching and checking the rowid (INDEX_NEXT_FIND_COST). (main.partition_pruning heap.heap_btree main.log_state) - Added INDEX_NEXT_FIND_COST to Range_rowid_filter_cost_info::lookup_cost to account of the time to find and check the next key value against the container - Introduced ha_keyread_time(rows) that takes into account finding the next row and copying the key value to 'record' (INDEX_COPY_COST). - Introduced ha_key_scan_time() for calculating an index scan over all rows. - Added IDX_LOOKUP_COST to keyread_time() as a startup cost. - Added index_only_fetch_cost() as a convenience function to OPT_RANGE. - keyread_time() cost is slightly reduced to prefer shorter keys. (main.index_merge_myisam) - All of the above caused some index_merge combinations to be rejected because of cost (main.index_intersect). In some cases 'ref' where replaced with index_merge because of the low cost calculation of get_sweep_read_cost(). - Some index usage moved from PRIMARY to a covering index. (main.subselect_innodb) - Changed cost calculation of filter to take INDEX_LOOKUP_COST and TIME_FOR_COMPARE into account. See sql_select.cc::apply_filter(). filter parameters and costs are now written to optimizer_trace. - Don't use matchings_records_in_range() to try to estimate the number of filtered rows for ranges. The reason is that we want to ensure that 'range' is calculated similar to 'ref'. There is also more work needed to calculate the selectivity when using ranges and ranges and filtering. This causes filtering column in EXPLAIN EXTENDED to be 100.00 for some cases where range cannot use filtering. (main.rowid_filter) - Introduced ha_scan_time() that takes into account the CPU cost of finding the next row and copying the row from the engine to 'record'. This causes costs of table scan to slightly increase and some test to changed their plan from ALL to RANGE or ALL to ref. (innodb.innodb_mysql, main.select_pkeycache) In a few cases where scan time of very small tables have lower cost than a ref or range, things changed from ref/range to ALL. (main.myisam, main.func_group, main.limit_rows_examined, main.subselect2) - Introduced ha_scan_and_compare_time() which is like ha_scan_time() but also adds the cost of the where clause (TIME_FOR_COMPARE). - Added small cost for creating temporary table for materialization. This causes some very small tables to use scan instead of materialization. - Added checking of the WHERE clause (TIME_FOR_COMPARE) of the accepted rows to ROR costs in get_best_ror_intersect() - Removed '- 0.001' from 'join->best_read' and optimize_straight_join() to ensure that the 'Last_query_cost' status variable contains the same value as the one that was calculated by the optimizer. - Take avg_io_cost() into account in handler::keyread_time() and handler::read_time(). This should have no effect as it's 1.0 by default, except for heap that overrides these functions. - Some 'ref_or_null' accesses changed to 'range' because of cost adjustments (main.order_by) - Added scan type "scan_with_join_cache" for optimizer_trace. This is just to show in the trace what kind of scan was used. - When using 'scan_with_join_cache' take into account number of preceding tables (as have to restore all fields for all previous table combination when checking the where clause) The new cost added is: (row_combinations * RECORD_COPY_COST * number_of_cached_tables). This increases the cost of join buffering in proportion of the number of tables in the join buffer. One effect is that full scans are now done earlier as the cost is then smaller. (main.join_outer_innodb, main.greedy_optimizer) - Removed the usage of 'worst_seeks' in cost_for_index_read as it caused wrong plans to be created; It prefered JT_EQ_REF even if it would be much more expensive than a full table scan. A related issue was that worst_seeks only applied to full lookup, not to clustered or index only lookups, which is not consistent. This caused some plans to use index scan instead of eq_ref (main.union) - Changed federated block size from 4096 to 1500, which is the typical size of an IO packet. - Added costs for reading rows to Federated. Needed as there is no caching of rows in the federated engine. - Added ha_innobase::rndpos_time() cost function. - A lot of extra things added to optimizer trace - More costs, especially for materialization and index_merge. - Make lables more uniform - Fixed a lot of minor bugs - Added 'trace_started()' around a lot of trace blocks. - When calculating ORDER BY with LIMIT cost for using an index the cost did not take into account the number of row retrivals that has to be done or the cost of comparing the rows with the WHERE clause. The cost calculated would be just a fraction of the real cost. Now we calculate the cost as we do for ranges and 'ref'. - 'Using index for group-by' is used a bit as we now take into account the WHERE clause cost when comparing with 'ref' and prefer the method with fewer row combinations. (main.group_min_max). Bugs fixed: - Fixed that we don't calculate TIME_FOR_COMPARE twice for some plans, like in optimize_straight_join() and greedy_search() - Fixed bug in save_explain_data where we could test for the wrong index when displaying 'Using index'. This caused some old plans to show 'Using index'. (main.subselect_innodb, main.subselect2) - Fixed bug in get_best_ror_intersect() where 'min_cost' was not updated, and the cost we compared with was not the one that was used. - Fixed very wrong cost calculation for priority queues in check_if_pq_applicable(). (main.order_by now correctly uses priority queue) - When calculating cost of EQ_REF or REF, we added the cost of comparing the WHERE clause with the found rows, not all row combinations. This made ref and eq_ref to be regarded way to cheap compared to other access methods. - FORCE INDEX cost calculation didn't take into account clustered or covered indexes. - JT_EQ_REF cost was estimated as avg_io_cost(), which is half the cost of a JT_REF key. This may be true for InnoDB primary key, but not for other unique keys or other engines. Now we use handler function to calculate the cost, which allows us to handle consistently clustered, covered keys and not covered keys. - ha_start_keyread() didn't call extra_opt() if keyread was already enabled but still changed the 'keyread' variable (which is wrong). Fixed by not doing anything if keyread is already enabled. - multi_range_read_info_cost() didn't take into account io_cost when calculating the cost of ranges. - fix_semijoin_strategies_for_picked_join_order() used the wrong record_count when calling best_access_path() for SJ_OPT_FIRST_MATCH and SJ_OPT_LOOSE_SCAN. - Hash joins didn't provide correct best_cost to the upper level, which means that the cost for hash_joins more expensive than calculated in best_access_path (a difference of 10x * TIME_OF_COMPARE). This is fixed in the new code thanks to that we now include TIME_OF_COMPARE cost in 'read_time'. Other things: - Added some 'if (thd->trace_started())' to speed up code - Removed not used function Cost_estimate::is_zero() - Simplified testing of HA_POS_ERROR in get_best_ror_intersect(). (No cost changes) - Moved ha_start_keyread() from join_read_const_table() to join_read_const() to enable keyread for all types of JT_CONST tables. - Made a few very short functions inline in handler.h Notes: - In main.rowid_filter the join order of order and lineitem is swapped. This is because the cost of doing a range fetch of lineitem(98 rows) is almost as big as the whole join of order,lineitem. The filtering will also ensure that we only have to do very small key fetches of the rows in lineitem. - main.index_merge_myisam had a few changes where we are now using less keys for index_merge. This is because index scans are now more expensive than before. - handler->optimizer_cache_cost is updated in ha_external_lock(). This ensures that it is up to date per statements. Not an optimal solution (for locked tables), but should be ok for now. - 'DELETE FROM t1 WHERE t1.a > 0 ORDER BY t1.a' does not take cost of filesort into consideration when table scan is chosen. (main.myisam_explain_non_select_all) - perfschema.table_aggregate_global_* has changed because an update on a table with 1 row will now use table scan instead of key lookup. TODO in upcomming commits: - Fix selectivity calculation for ranges with and without filtering and when there is a ref access but scan is chosen. For this we have to store the lowest known value for 'accepted_records' in the OPT_RANGE structure. - Change that records_read does not include filtered rows. - test_if_cheaper_ordering() needs to be updated to properly calculate costs. This will fix tests like main.order_by_innodb, main.single_delete_update - Extend get_range_limit_read_cost() to take into considering cost_for_index_read() if there where no quick keys. This will reduce the computed cost for ORDER BY with LIMIT in some cases. (main.innodb_ext_key) - Fix that we take into account selectivity when counting the number of rows we have to read when considering using a index table scan to resolve ORDER BY. - Add new calculation for rndpos_time() where we take into account the benefit of reading multiple rows from the same page.	4 years ago
Monty	38058c04a4	MDEV-26585 Wrong query results when `using index for group-by` The problem was that "group_min_max optimization" does not work if some aggregate functions, like COUNT(), is used. The function get_best_group_min_max() is using the join->sum_funcs array to check which aggregate functions are used. The bug was that aggregates in HAVING where not yet added to join->sum_funcs at the time get_best_group_min_max() was called. Fixed by populate join->sum_funcs already in prepare, which means that all sum functions will be in join->sum_funcs in get_best_group_min_max(). A benefit of this approach is that we can remove several calls to make_sum_func_list() from the code and simplify the function. I removed some wrong setting of 'sort_and_group'. This variable is set when alloc_group_fields() is called, as part of allocating the cache needed by end_send_group() and does not need to be set by other functions. One problematic thing was that Spider is using join->sum_funcs to detect at which stage the optimizer is and do internal calculations of aggregate functions. Updating join->sum_funcs early caused Spider to fail when trying to find min/max values in opt_sum_query(). Fixed by temporarily resetting sum_funcs during opt_sum_query(). Reviewer: Sergei Petrunia	4 years ago
Igor Babaev	0041265671	MDEV-27510 Query returns wrong result when using split optimization This bug may affect the queries that uses a grouping derived table with grouping list containing references to columns from different tables if the optimizer decides to employ the split optimization for the derived table. In some very specific cases it may affect queries with a grouping derived table that refers only one base table. This bug was caused by an improper fix for the bug MDEV-25128. The fix tried to get rid of the equality conditions pushed into the where clause of the grouping derived table T to which the split optimization had been applied. The fix erroneously assumed that only those pushed equalities that were used for ref access of the tables referenced by T were needed. In fact the function remove_const() that figures out what columns from the group list can be removed if the split optimization is applied can uses other pushed equalities as well. This patch actually provides a proper fix for MDEV-25128. Rather than trying to remove invalid pushed equalities referencing the fields of SJM tables with a look-up access the patch attempts not to push such equalities. Approved by Oleksandr Byelkin <sanja@mariadb.com>	4 years ago
Monty	cc125bebfe	Fix all warnings given by UBSAN The 'special' cases where we disable, suppress or circumvent UBSAN are: - ref10 source (as here we intentionally do some shifts that UBSAN complains about. - x86 version of optimized int#korr() methods. UBSAN do not like unaligned memory access of integers. Fixed by using byte_order_generic.h when compiling with UBSAN - We use smaller thread stack with ASAN and UBSAN, which forced me to disable a few tests that prints the thread stack size. - Verifying class types does not work for shared libraries. I added suppression in mysql-test-run.pl for this case. - Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is safe to have overflows (two cases, in item_func.cc). Things fixed: - Don't left shift signed values (byte_order_generic.h, mysqltest.c, item_sum.cc and many more) - Don't assign not non existing values to enum variables. - Ensure that bool and enum values are properly initialized in constructors. This was needed as UBSAN checks that these types has correct values when one copies an object. (gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...) - Ensure we do not called handler functions on unallocated objects or deleted objects. (events.cc, sql_acl.cc). - Fixed bugs in Item_sp::Item_sp() where we did not call constructor on Query_arena object. - Fixed several cast of objects to an incompatible class! (Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc, sql_select.cc ...) - Ensure we do not do integer arithmetic that causes over or underflows. This includes also ++ and -- of integers. (Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...) - Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that value_type is initialized to this instead of to -1, which is not a valid enum value for json_value_types. - Ensure we do not call memcpy() when second argument could be null. Other things: - Changed struct st_position to an OBJECT and added an initialization function to it to ensure that we do not copy or use uninitialized members. The change to a class was also motived that we used "struct st_position" and POSITION randomly trough the code which was confusing. - Notably big rewrite in sql_acl.cc to avoid using deleted objects. - Changed in sql_partition to use '^' instead of '-'. This is safe as the operator is either 0 or 0x8000000000000000ULL. - Added check for select_nr < INT_MAX in JOIN::build_explain() to avoid bug when get_select() could return NULL. - Reordered elements in POSITION for better alignment. - Changed sql_test.cc::print_plan() to use pointers instead of objects. - Fixed bug in find_set() where could could execute '1 << -1'. - Added variable have_sanitizer, used by mtr. (This variable was before only in 10.5 and up). It can now have one of two values: ASAN or UBSAN. - Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked it virtual. This was an effort to get UBSAN to work with loaded storage engines. I kept the change as the new place is better. - Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast in tabutil.cpp. Changes that should not be needed but had to be done to suppress warnings from UBSAN: - Added static_cast<<uint16_t>> around shift to get rid of a LOT of compiler warnings when using UBSAN. - Had to change some '/' of 2 base integers to shift to get rid of some compile time warnings. Fixes: MDEV-25505 Assertion `old_flags == ((my_flags & 0x10000U) ? 1 : 0) fixed (was caused by an old version if this commit). Reviewed by: - Json changes: Alexey Botchkov - Charset changes in ctype-uca.c: Alexander Barkov - InnoDB changes: Marko Mäkelä - sql_acl.cc changes: Vicențiu Ciorbaru - build_explain() changes: Sergey Petrunia Temporary commit to log changes for UBSAN	5 years ago
Monty	db9398ba5e	Improved code comment and removed nop test	5 years ago
Monty	08bc062e3c	Remove some usage of Check_level_instant_set and Sql_mode_save The reason for the removal are: - Generates more code - Storing and retreving THD - Causes extra code and daata to be generated to handle possible throw exceptions (which never happens in MariaDB code) - Uses more stack space Other things: - Changed convert_const_to_int() to use item->save_in_field_no_warnings(), which made the code shorter and simpler. - Removed not needed code in Sp_handler::sp_create_routine() - Added thd as argument to store_key.copy() to make function simpler - Added thd as argument to some subselect* constructor that inherites from Item_subselect.	5 years ago
Monty	be093c81a7	MDEV-24089 support oracle syntax: rownum The ROWNUM() function is for SELECT mapped to JOIN->accepted_rows, which is incremented for each accepted rows. For Filesort, update, insert, delete and load data, we map ROWNUM() to internal variables incremented when the table is changed. The connection between the row counter and Item_func_rownum is done in sql_select.cc::fix_items_after_optimize() and sql_insert.cc::fix_rownum_pointers() When ROWNUM() is used anywhere in query, the optimization to ignore ORDER BY in sub queries are disabled. This was done to get the following common Oracle query to work: select * from (select * from t1 order by a desc) as t where rownum() <= 2; MDEV-3926 "Wrong result with GROUP BY ... WITH ROLLUP" contains a discussion about this topic. LIMIT optimization is enabled when in a top level WHERE clause comparing ROWNUM() with a numerical constant using any of the following expressions: - ROWNUM() < # - ROWNUM() <= # - ROWNUM() = 1 ROWNUM() can be also be the right argument to the comparison function. LIMIT optimization is done in two cases: - For the current sub query when the ROWNUM comparison is done on the top level: SELECT * from t1 WHERE rownum() <= 2 AND t1.a > 0 - For an inner sub query, when the upper level has only a ROWNUM comparison in the WHERE clause: SELECT * from (select * from t1) as t WHERE rownum() <= 2 In Oracle mode, one can also use ROWNUM without parentheses. Other things: - Fixed bug where the optimizer tries to optimize away sub queries with RAND_TABLE_BIT set (non-deterministic queries). Now these sub queries will not be converted to joins. This bug fix was also needed to get rownum() working inside subqueries. - In remove_const() remove setting simple_order to FALSE if ROLLUP is USED. This code was disable a long time ago because of wrong assignment in the following code. Instead we set simple_order to false if RAND_TABLE_BIT was used in the SELECT list. This ensures that we don't delete ORDER BY if the result set is not deterministic, like in 'SELECT RAND() AS 'r' FROM t1 ORDER BY r'; - Updated parameters for Sort_param::init_for_filesort() to be able to provide filesort with information where the number of accepted rows should be stored - Reordered fields in class Filesort to optimize storage layout - Added new error messsage to tell that a function can't be used in HAVING - Added field 'with_rownum' to THD to mark that ROWNUM() is used in the query. Co-author: Oleksandr Byelkin <sanja@mariadb.com> LIMIT optimization for sub query	5 years ago
Monty	30f0a246a0	Added override to all releveant methods in Item (and a few other classes) Other things: - Remove inline and virtual for methods that are overrides - Added a 'final' to some Item classes	5 years ago
Michael Widenius	3105c9e7a5	Change bitfields in Item to an uint16 The reason for the change is that neither clang or gcc can do efficient code when several bit fields are change at the same time or when copying one or more bits between identical bit fields. Updated bits explicitely with & and \| is MUCH more efficient than what current compilers can do.	5 years ago
Sergei Petrunia	2820f30dde	MDEV-23723: Crash when test_if_skip_sort_order() is checked for derived ... The problem was caused by the following scenario: Subquery's table has two indexes, KEY a(a), KEY a_b(a,b) - LATERAL DERIVED optimization decides to use index a. = The subquery uses ref access over key a. - test_if_skip_sort_order() sees that KEY a_b satisfies the subquery's GROUP BY clause, and attempts to switch to it. = It fails to do so, because KEYUSE objects for index a_b are switched off. Fixed by disallowing to change the ref access key if it uses KEYUSE objects injected by LATERAL DERIVED optimization.	5 years ago
Vicențiu Ciorbaru	299b935320	MDEV-23908: Implement SELECT ... OFFSET ... FETCH ... This commit implements the standard SQL extension OFFSET start { ROW \| ROWS } [FETCH { FIRST \| NEXT } [ count ] { ROW \| ROWS } { ONLY \| WITH TIES }] To achieve this a reserved keyword OFFSET is introduced. The general logic for WITH TIES implies: 1. The number of rows a query returns is no longer known during optimize phase. Adjust optimizations to no longer consider this. 2. During end_send make use of an "order Cached_item"to compare if the ORDER BY columns changed. Keep returning rows until there is a change. This happens only after we reached the row limit. 3. Within end_send_group, the order by clause was eliminated. It is still possible to keep the optimization of using end_send_group for producing the final result set.	5 years ago
Alexey Botchkov	e9fd327ee3	MDEV-17399 Add support for JSON_TABLE. The specific table handler for the table functions was introduced, and used to implement JSON_TABLE.	5 years ago
Monty	031f11717d	Fix all warnings given by UBSAN The easiest way to compile and test the server with UBSAN is to run: ./BUILD/compile-pentium64-ubsan and then run mysql-test-run. After this commit, one should be able to run this without any UBSAN warnings. There is still a few compiler warnings that should be fixed at some point, but these do not expose any real bugs. The 'special' cases where we disable, suppress or circumvent UBSAN are: - ref10 source (as here we intentionally do some shifts that UBSAN complains about. - x86 version of optimized int#korr() methods. UBSAN do not like unaligned memory access of integers. Fixed by using byte_order_generic.h when compiling with UBSAN - We use smaller thread stack with ASAN and UBSAN, which forced me to disable a few tests that prints the thread stack size. - Verifying class types does not work for shared libraries. I added suppression in mysql-test-run.pl for this case. - Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is safe to have overflows (two cases, in item_func.cc). Things fixed: - Don't left shift signed values (byte_order_generic.h, mysqltest.c, item_sum.cc and many more) - Don't assign not non existing values to enum variables. - Ensure that bool and enum values are properly initialized in constructors. This was needed as UBSAN checks that these types has correct values when one copies an object. (gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...) - Ensure we do not called handler functions on unallocated objects or deleted objects. (events.cc, sql_acl.cc). - Fixed bugs in Item_sp::Item_sp() where we did not call constructor on Query_arena object. - Fixed several cast of objects to an incompatible class! (Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc, sql_select.cc ...) - Ensure we do not do integer arithmetic that causes over or underflows. This includes also ++ and -- of integers. (Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...) - Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that value_type is initialized to this instead of to -1, which is not a valid enum value for json_value_types. - Ensure we do not call memcpy() when second argument could be null. - Fixed that Item_func_str::make_empty_result() creates an empty string instead of a null string (safer as it ensures we do not do arithmetic on null strings). Other things: - Changed struct st_position to an OBJECT and added an initialization function to it to ensure that we do not copy or use uninitialized members. The change to a class was also motived that we used "struct st_position" and POSITION randomly trough the code which was confusing. - Notably big rewrite in sql_acl.cc to avoid using deleted objects. - Changed in sql_partition to use '^' instead of '-'. This is safe as the operator is either 0 or 0x8000000000000000ULL. - Added check for select_nr < INT_MAX in JOIN::build_explain() to avoid bug when get_select() could return NULL. - Reordered elements in POSITION for better alignment. - Changed sql_test.cc::print_plan() to use pointers instead of objects. - Fixed bug in find_set() where could could execute '1 << -1'. - Added variable have_sanitizer, used by mtr. (This variable was before only in 10.5 and up). It can now have one of two values: ASAN or UBSAN. - Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked it virtual. This was an effort to get UBSAN to work with loaded storage engines. I kept the change as the new place is better. - Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast in tabutil.cpp. - Added HAVE_REPLICATION around usage of rgi_slave, to get embedded server to compile with UBSAN. (Patch from Marko). - Added #ifdef for powerpc64 to avoid a bug in old gcc versions related to integer arithmetic. Changes that should not be needed but had to be done to suppress warnings from UBSAN: - Added static_cast<<uint16_t>> around shift to get rid of a LOT of compiler warnings when using UBSAN. - Had to change some '/' of 2 base integers to shift to get rid of some compile time warnings. Reviewed by: - Json changes: Alexey Botchkov - Charset changes in ctype-uca.c: Alexander Barkov - InnoDB changes & Embedded server: Marko Mäkelä - sql_acl.cc changes: Vicențiu Ciorbaru - build_explain() changes: Sergey Petrunia	5 years ago
Nikita Malyavin	21809f9a45	MDEV-17556 Assertion `bitmap_is_set_all(&table->s->all_set)' failed The assertion failed in handler::ha_reset upon SELECT under READ UNCOMMITTED from table with index on virtual column. This was the debug-only failure, though the problem is mush wider: * MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw bitmap. * read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP * The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE * The pointers to the stored MY_BITMAPs, like orig_read_set etc, and sometimes all_set and tmp_set, are assigned to the pointers. * Sometimes tmp_use_all_columns is used to substitute the raw bitmap directly with all_set.bitmap * Sometimes even bitmaps are directly modified, like in TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called. The last three bullets in the list, when used together (which is mostly always) make the program flow cumbersome and impossible to follow, notwithstanding the errors they cause, like this MDEV-17556, where tmp_set pointer was assigned to read_set, write_set and vcol_set, then its bitmap was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call, and then bitmap_clear_all(&tmp_set) was applied to all this. To untangle this knot, the rule should be applied: * Never substitute bitmaps! This patch is about this. orig_, all_set bitmaps are never substituted already. This patch changes the following function prototypes: tmp_use_all_columns, dbug_tmp_use_all_columns to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map* * tmp_restore_column_map, dbug_tmp_restore_column_maps to accept MY_BITMAP* instead of my_bitmap_map* These functions now will substitute read_set/write_set/vcol_set directly, and won't touch underlying bitmaps.	5 years ago
Nikita Malyavin	e25623e78a	MDEV-17556 Assertion `bitmap_is_set_all(&table->s->all_set)' failed The assertion failed in handler::ha_reset upon SELECT under READ UNCOMMITTED from table with index on virtual column. This was the debug-only failure, though the problem is mush wider: * MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw bitmap. * read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP * The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE * The pointers to the stored MY_BITMAPs, like orig_read_set etc, and sometimes all_set and tmp_set, are assigned to the pointers. * Sometimes tmp_use_all_columns is used to substitute the raw bitmap directly with all_set.bitmap * Sometimes even bitmaps are directly modified, like in TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called. The last three bullets in the list, when used together (which is mostly always) make the program flow cumbersome and impossible to follow, notwithstanding the errors they cause, like this MDEV-17556, where tmp_set pointer was assigned to read_set, write_set and vcol_set, then its bitmap was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call, and then bitmap_clear_all(&tmp_set) was applied to all this. To untangle this knot, the rule should be applied: * Never substitute bitmaps! This patch is about this. orig_, all_set bitmaps are never substituted already. This patch changes the following function prototypes: tmp_use_all_columns, dbug_tmp_use_all_columns to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map* * tmp_restore_column_map, dbug_tmp_restore_column_maps to accept MY_BITMAP* instead of my_bitmap_map* These functions now will substitute read_set/write_set/vcol_set directly, and won't touch underlying bitmaps.	5 years ago
Sergei Golubchik	00f54b56b1	cleanup: RAII helper for changing thd->count_cuted_rows	5 years ago
Alexander Barkov	080522dcd7	MDEV-23825 Join select_handler and Pushdown_select + XPand changes	5 years ago
Varun Gupta	1e31d74833	MDEV-17066: Bytes lost or Assertion `status_var.local_memory_used == 0 after DELETE with subquery with ROLLUP The issue here is when records are read from the temporary file (filesort result in this case) via a cache(rr_from_cache). The cache is initialized with init_rr_cache. For correlated subquery the cache allocation is happening at each execution of the subquery but the deallocation happens only once and that was when the query execution was done. So generally for subqueries we do two types of cleanup 1) Full cleanup: we should free all resources of the query(like temp tables). This is done generally when the query execution is complete or the subquery re-execution is not needed (case with uncorrelated subquery) 2) Partial cleanup: Minor cleanup that is required if the subquery needs recalculation. This is done for all the structures that need to be allocated for each execution (example SORT_INFO for filesort is allocated for each execution of the correlated subquery). The fix here would be free the cache used by rr_from_cache in the partial cleanup phase.	5 years ago
Varun Gupta	7148b84673	MDEV-13694: Wrong result upon GROUP BY with orderby_uses_equalities=on For the case when the SJM scan table is the first table in the join order, then if we want to do the sorting on the SJM scan table, then we need to make sure that we unpack the values to base table fields in two cases: 1) Reading the SJM table and writing the sort-keys inside the sort-buffer 2) Reading the sorted data from the sort file	5 years ago
Varun Gupta	648b54746c	MDEV-22399: Remove multiple calls to enable and disable Handler::keyread and perform it after the plan refinement phase is done Introduce a function to enable keyreads for indexes and use this function when all the decision of plan refinement phase are done.	5 years ago
Varun Gupta	4a6b28c7b9	MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed. A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC is used and there is no window function, then there is no need to create a temporary table as there is no stage to compute WINDOW FUNCTION	6 years ago
Eugene Kosov	fb74de9728	MDEV-22006 runtime error: null pointer passed as argument 2, which is declared to never be null in JOIN::copy_ref_ptr_array() Do not memcpy() a zero-length buffer	6 years ago
Sergei Golubchik	0d837e8153	perfschema table io instrumentation related changes	6 years ago
Sergei Golubchik	7c58e97bf6	perfschema memory related instrumentation changes	6 years ago
Sergei Golubchik	2ac3121af2	perfschema - various collateral cleanups and small changes	6 years ago
Varun Gupta	891609b571	MDEV-21318: Wrong results with window functions and implicit grouping The issue here is for degenerate joins we should execute the window function but it is not getting executed in all the cases. To get the window function values window function needs to be executed always. This currently does not happen in few cases where the join would return 0 or 1 row like 1) IMPOSSIBLE WHERE 2) MIN/MAX optimization 3) EMPTY CONST TABLE The fix is to make sure that window functions get executed and the temporary table is setup for the execution of window functions	6 years ago
Sergei Golubchik	a4a025f5d1	cleanup: don't pass wild_num to setup_wild() because internally setup_wild() adjusts select_lex->with_wild directly anyway, so there is no reason to pretend that the number of '*' may be anything else but select_lex->with_wild And don't update select_lex->item_list, because fields can come from anywhere and don't necessarily have to be copied into select_lex.	6 years ago
Sergei Petrunia	c8dc866fde	MDEV-20371: Invalid reads at plan refinement stage: join->positions... best_access_path() is called from two optimization phases: 1. Plan choice phase, in choose_plan(). Here, the join prefix being considered is in join->positions[] 2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order Here, the join prefix is in join->best_positions[] It used to access join->positions[] from stage #2. This didnt cause any valgrind or asan failures (as join->positions[] has been written-to before) but the effect was similar to that of reading the random data: The join prefix we've picked (in join->best_positions) could have nothing in common with the join prefix that was last to be considered (in join->positions).	6 years ago
Igor Babaev	9380850d87	MDEV-15777 Use inferred IS NOT NULL predicates in the range optimizer This patch introduces the optimization that allows range optimizer to consider index range scans that are built employing NOT NULL predicates inferred from WHERE conditions and ON expressions. The patch adds a new optimizer switch not_null_range_scan.	6 years ago
Aleksey Midenkov	6dd3f24090	MDEV-19740 Debug build of 10.3.15 FTBFS * Replace LINT_INIT for non-struct types with ctor initializers; * Check BUILD_DEPS list is not empty so REMOVE_DUPLICATES won't throw error.	6 years ago
Alexander Barkov	afe6eb499d	Revert "MDEV-20342 Turn Field::flags from a member to a method" This reverts commit `e86010f909`. Reverting on Monty's request, as this change makes merging things from 10.5 to 10.2 much harder.	6 years ago
Alexander Barkov	e86010f909	MDEV-20342 Turn Field::flags from a member to a method	6 years ago
Alexander Barkov	ac93d7d674	MDEV-19593 Split create_schema_table() into virtual methods in Type_handler	7 years ago
Vicențiu Ciorbaru	5543b75550	Update FSF Address * Update wrong zip-code	7 years ago
Galina Shalygina	ae15f91f22	MDEV-18769 Assertion `fixed == 1' failed in Item_cond_or::val_int This bug is caused by pushdown from HAVING into WHERE. It appears because condition that is pushed wasn't fixed. It is also discovered that condition pushdown from HAVING into WHERE is done wrong. There is no need to build clones for some conditions that can be pushed. They can be simply moved from HAVING into WHERE without cloning. build_pushable_cond_for_having_pushdown(), remove_pushed_top_conjuncts_for_having() methods are changed. It is found that there is no transformation made for fields of pushed condition. field_transformer_for_having_pushdown transformer is added. New tests are added. Some comments are changed.	7 years ago
Sergei Golubchik	f1134d5676	post-merge: gcc 8 warnings note: Inherit String from Sql_alloc, to get operators new and new[] in sync in rocksdb gcc was complaining that non-lvalue was cast to const.	7 years ago
Sergei Golubchik	3d2d060b62	fix gcc 8 compiler warnings There were two newly enabled warnings: 1. cast for a function pointers. Affected sql_analyse.h, mi_write.c and ma_write.cc, mf_iocache-t.cc, mysqlbinlog.cc, encryption.cc, etc 2. memcpy/memset of nontrivial structures. Fixed as: * the warning disabled for InnoDB * TABLE, TABLE_SHARE, and TABLE_LIST got a new method reset() which does the bzero(), which is safe for these classes, but any other bzero() will still cause a warning * Table_scope_and_contents_source_st uses `TABLE_LIST ` (trivial) instead of `SQL_I_List<TABLE_LIST>` (not trivial) so it's safe to bzero now. added casts in debug_sync.cc and sql_select.cc (for JOIN) * move assignment method for MDL_request instead of memcpy() * PARTIAL_INDEX_INTERSECT_INFO::init() instead of bzero() * remove constructor from READ_RECORD() to make it trivial * replace some memcpy() with c++ copy assignments	7 years ago
Igor Babaev	8f4de38f65	MDEV-18467 Server crashes in fix_semijoin_strategies_for_picked_join_order If a splittable materialized derived table / view T is used in a inner nest of an outer join with impossible ON condition then T is marked as a constant table. Yet the execution plan to build T is still searched for in spite of the fact that is not needed. So it should be set.	7 years ago
Galina Shalygina	7a77b221f1	MDEV-7486: Condition pushdown from HAVING into WHERE Condition can be pushed from the HAVING clause into the WHERE clause if it depends only on the fields that are used in the GROUP BY list or depends on the fields that are equal to grouping fields. Aggregate functions can't be pushed down. How the pushdown is performed on the example: SELECT t1.a,MAX(t1.b) FROM t1 GROUP BY t1.a HAVING (t1.a>2) AND (MAX(c)>12); => SELECT t1.a,MAX(t1.b) FROM t1 WHERE (t1.a>2) GROUP BY t1.a HAVING (MAX(c)>12); The implementation scheme: 1. Extract the most restrictive condition cond from the HAVING clause of the select that depends only on the fields that are used in the GROUP BY list of the select (directly or indirectly through equalities) 2. Save cond as a condition that can be pushed into the WHERE clause of the select 3. Remove cond from the HAVING clause if it is possible The optimization is implemented in the function st_select_lex::pushdown_from_having_into_where(). New test file having_cond_pushdown.test is created.	7 years ago
Igor Babaev	16327fc2e7	MDEV-17096 Pushdown of simple derived tables to storage engines MDEV-17631 select_handler for a full query pushdown Interfaces + Proof of Concept for federatedx with test cases. The interfaces have been developed for integration of ColumnStore engine.	7 years ago
Igor Babaev	658128af43	MDEV-16188 Use in-memory PK filters built from range index scans This patch contains a full implementation of the optimization that allows to use in-memory rowid / primary filters built for range conditions over indexes. In many cases usage of such filters reduce the number of disk seeks spent for fetching table rows. In this implementation the choice of what possible filter to be applied (if any) is made purely on cost-based considerations. This implementation re-achitectured the partial implementation of the feature pushed by Galina Shalygina in the commit `8d5a11122c`. Besides this patch contains a better implementation of the generic handler function handler::multi_range_read_info_const() that takes into account gaps between ranges when calculating the cost of range index scans. It also contains some corrections of the implementation of the handler function records_in_range() for MyISAM. This patch supports the feature for InnoDB and MyISAM.	7 years ago
Sergei Petrunia	3b6d903852	MDEV-17493: Partition pruning doesn't work for nested outer joins Reuse the fix for MDEV-17518 here, too.	7 years ago
Galina Shalygina	8d5a11122c	MDEV-16188: Use in-memory PK filters built from range index scans First phase: make optimizer choose to use filter and show it in EXPLAIN.	7 years ago
Alexander Barkov	637af78383	MDEV-16309 Split ::create_tmp_field() into virtual methods in Item Detailed: changes: 1. Moving Field specific code into new methods on Field: - Field *Field::create_tmp_field(...) - virtual void init_for_tmp_table(...) 2. Removing virtual Item::create_tmp_field(). Adding instead a new virtual method Item::create_tmp_field_ex(). Note, a virtual create_tmp_field() still exists, but only for Item_sum. This resembles 10.0 code structure. Perhaps create_tmp_field() should be removed from Item_sum and Item_sum descendants should override create_tmp_field_ex() directly. This can be done in a separate commit. 3. Adding helper classes Tmp_field_src and Tmp_field_param, to make the API for Item::create_tmp_field_ex() smaller and easier to extend in the future. 4. Decomposing the public function create_tmp_field() into virtual implementations for Item and a number of its descendants: - Item_basic_value - Item_sp_variable - Item_name_const - Item_result_field - Item_field - Item_ref - Item_type_holder - Item_row - Item_func_sp - Item_func_user_var - Item_sum - Item_sum_field - Item_proc 5. Adding DBUG_ASSERT-only virtual implementations for Item types that should not appear in create_tmp_table_ex(), for easier debugging: - Item_nodeset_func - Item_nodeset_to_const_comparator - Item_null_result - Item_copy - Item_ident_for_show - Item_user_var_as_out_param 6. Moving public function create_tmp_field_from_field() as a method to Item_field. 7. Removing Item::set_result_field(). It's not needed any more. 8. Cleanup: Removing the enum value "EXPR_CACHE_ITEM", as it's not used for a very long time.	8 years ago
Sergei Golubchik	28dbdf3d79	MDEV-14551 Can't find record in table on multi-table update with ORDER BY preserve positions if the multi-update join is using tmp table: * store positions in the tmp table if needed JOIN::add_fields_for_current_rowid() * take positions from the tmp table, not from file->position(): multi_update::prepare2()	8 years ago
Sergei Golubchik	e17e798599	cleanup: simplify multi-update's juggling with positions introduce Item_temptable_rowid() that is used to store table->file->position() in the temporary table record	8 years ago
Sergei Golubchik	c368878fac	cleanup: comments and whitespaces	8 years ago
Galina Shalygina	d3ff133390	MDEV-12387 Push conditions into materialized subqueries The logic and the implementation scheme are similar with the MDEV-9197 Pushdown conditions into non-mergeable views/derived tables How the push down is made on the example: select * from t1 where a>3 and b>10 and (a,b) in (select x,max(y) from t2 group by x); --> select * from t1 where a>3 and b>10 and (a,b) in (select x,max(y) from t2 where x>3 group by x having max(y)>10); The implementation scheme: 1. Search for the condition cond that depends only on the fields from the left part of the IN subquery (left_part) 2. Find fields F_group in the select of the right part of the IN subquery (right_part) that are used in the GROUP BY 3. Extract from the cond condition cond_where that depends only on the fields from the left_part that stay at the same places in the left_part (have the same indexes) as the F_group fields in the projection of the right_part 4. Transform cond_where so it can be pushed into the WHERE clause of the right_part and delete cond_where from the cond 5. Transform cond so it can be pushed into the HAVING clause of the right_part The optimization is made in the Item_in_subselect::pushdown_cond_for_in_subquery() and is controlled by the variable condition_pushdown_for_subquery. New test file in_subq_cond_pushdown.test is created. There are also some changes made for setup_jtbm_semi_joins(). Now it is decomposed into the 2 procedures: setup_degenerate_jtbm_semi_joins() that is called before optimize_cond() for cond and setup_jtbm_semi_joins() that is called after optimize_cond(). New setup_jtbm_semi_joins() is made in the way so that the result of its work is the same as if it was called before optimize_cond(). The code that is common for pushdown into materialized derived and into materialized IN subqueries is factored out into pushdown_cond_for_derived(), Item_in_subselect::pushdown_cond_for_in_subquery() and st_select_lex::pushdown_cond_into_where_clause().	8 years ago

1 2 3 4 5 ...

826 Commits (1356b77011df38af9b3c31a4fe57647aa281de8a)