Variables | |
| unsigned int | current_vector_size |
Function Documentation
|
static |
PTR is a pointer to an array of type TYPE. Return a representation of *PTR. The memory reference replaces those in FIRST_DR (and its group).
References build_int_cst(), DR_REF, get_ptr_info(), reference_alias_ptr_type(), and set_ptr_info_alignment().
Referenced by vectorizable_load(), and vectorizable_store().
|
static |
Return a variable of type ELEM_TYPE[NELEMS].
References build_array_type_nelts(), and create_tmp_var().
Referenced by vectorizable_load(), and vectorizable_store().
|
static |
Function exist_non_indexing_operands_for_use_p USE is one of the uses attached to STMT. Check if USE is used in STMT for anything other than indexing an array.
References gimple_assign_copy_p(), gimple_assign_lhs(), gimple_assign_rhs1(), and vinfo_for_stmt().
Referenced by process_use().
| void free_stmt_vec_info | ( | ) |
Free stmt vectorization related info.
References free(), gsi_end_p(), gsi_next(), gsi_stmt(), set_vinfo_for_stmt(), si, and vinfo_for_stmt().
Referenced by destroy_bb_vec_info(), destroy_loop_vec_info(), free_stmt_vec_info_vec(), slpeel_make_loop_iterate_ntimes(), vect_remove_stores(), vect_schedule_slp(), and vect_transform_loop().
| void free_stmt_vec_info_vec | ( | void | ) |
Free hash table for stmt_vec_info.
References free_stmt_vec_info(), and stmt_vec_info_vec.
Referenced by execute_vect_slp(), parallelize_loops(), and vectorize_loops().
| tree get_same_sized_vectype | ( | ) |
Function get_same_sized_vectype Returns a vector type corresponding to SCALAR_TYPE of size VECTOR_TYPE if supported by the target.
References get_vectype_for_scalar_type_and_size().
Referenced by vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_operation(), and vectorizable_shift().
| tree get_vectype_for_scalar_type | ( | ) |
Function get_vectype_for_scalar_type. Returns the vector type corresponding to SCALAR_TYPE as supported by the target.
References current_vector_size, and get_vectype_for_scalar_type_and_size().
Referenced by check_bool_pattern(), get_initial_def_for_induction(), get_initial_def_for_reduction(), increase_alignment(), vect_analyze_data_refs(), vect_analyze_slp_instance(), vect_analyze_stmt(), vect_build_slp_tree_1(), vect_create_epilog_for_reduction(), vect_determine_vectorization_factor(), vect_gen_perm_mask(), vect_get_constant_vectors(), vect_get_vec_def_for_operand(), vect_model_reduction_cost(), vect_pattern_recog_1(), vect_recog_bool_pattern(), vect_recog_divmod_pattern(), vect_recog_mixed_size_cond_pattern(), vect_recog_over_widening_pattern(), vect_recog_pow_pattern(), vect_recog_rotate_pattern(), vect_recog_vector_vector_shift_pattern(), vect_recog_widen_mult_pattern(), vect_recog_widen_shift_pattern(), vect_supportable_shift(), and vect_transform_slp_perm_load().
|
static |
Function get_vectype_for_scalar_type_and_size. Returns the vector type corresponding to SCALAR_TYPE and SIZE as supported by the target.
References build_nonstandard_integer_type(), build_vector_type(), mode_for_vector(), targetm, lang_hooks_for_types::type_for_mode, and lang_hooks::types.
Referenced by get_same_sized_vectype(), and get_vectype_for_scalar_type().
| void init_stmt_vec_info_vec | ( | void | ) |
Create a hash table for stmt_vec_info.
References stmt_vec_info_vec.
Referenced by execute_vect_slp(), parallelize_loops(), and vectorize_loops().
| stmt_vec_info new_stmt_vec_info | ( | gimple | stmt, |
| loop_vec_info | loop_vinfo, | ||
| bb_vec_info | bb_vinfo | ||
| ) |
Function new_stmt_vec_info. Create and initialize a new stmt_vec_info struct for STMT.
References is_loop_header_bb_p(), loop_vect, undef_vec_info_type, vect_internal_def, vect_unknown_def_type, and vect_unused_in_scope.
Referenced by get_initial_def_for_induction(), new_bb_vec_info(), new_loop_vec_info(), vect_create_data_ref_ptr(), vect_create_epilog_for_reduction(), vect_finish_stmt_generation(), vect_is_simple_reduction_1(), vect_mark_pattern_stmts(), vect_recog_bool_pattern(), vect_recog_divmod_pattern(), vect_recog_mixed_size_cond_pattern(), vect_recog_rotate_pattern(), vectorizable_load(), and vectorizable_reduction().
|
static |
Given a vector type VECTYPE returns the VECTOR_CST mask that implements reversal of the vector elements. If that is impossible to do, returns NULL.
References vect_gen_perm_mask().
Referenced by vectorizable_load().
|
static |
Given a vector variable X and Y, that was generated for the scalar STMT, generate instructions to permute the vector elements of X and Y using permutation mask MASK_VEC, insert them at *GSI and return the permuted vector variable.
References gimple_assign_lhs(), gimple_build_assign_with_ops(), make_ssa_name(), vect_create_destination_var(), and vect_finish_stmt_generation().
Referenced by vectorizable_load().
|
static |
References dump_enabled_p(), dump_printf_loc(), exist_non_indexing_operands_for_use_p(), flow_bb_inside_loop_p(), flow_loop_nested_p(), gimple_bb(), gimple_nop_p(), is_pattern_stmt_p(), basic_block_def::loop_father, vect_double_reduction_def, vect_is_simple_use(), vect_location, vect_mark_relevant(), vect_nested_cycle, vect_reduction_def, vect_unused_in_scope, vect_used_by_reduction, vect_used_in_outer, vect_used_in_outer_by_reduction, vect_used_in_scope, and vinfo_for_stmt().
Referenced by vect_mark_stmts_to_be_vectorized().
|
static |
ARRAY is an array of vectors created by create_vector_array. Return an SSA_NAME for the vector in index N. The reference is part of the vectorization of STMT and the vector is associated with scalar destination SCALAR_DEST.
References build_int_cst(), gimple_assign_set_lhs(), make_ssa_name(), vect_create_destination_var(), and vect_finish_stmt_generation().
Referenced by vectorizable_load().
| unsigned record_stmt_cost | ( | stmt_vector_for_cost * | body_cost_vec, |
| int | count, | ||
| enum vect_cost_for_stmt | kind, | ||
| stmt_vec_info | stmt_info, | ||
| int | misalign, | ||
| enum vect_cost_model_location | where | ||
| ) |
Record the cost of a statement, either by directly informing the target model or by saving it in a vector for later processing. Return a preliminary estimate of the statement's cost.
References add_stmt_cost(), add_stmt_info_to_vec(), builtin_vectorization_cost(), count, and stmt_vectype().
Referenced by vect_analyze_slp_cost_1(), vect_get_known_peeling_cost(), vect_get_load_cost(), vect_get_store_cost(), vect_model_load_cost(), vect_model_simple_cost(), and vect_model_store_cost().
| bool stmt_in_inner_loop_p | ( | ) |
Return TRUE iff the given statement is in an inner loop relative to the loop being vectorized.
References gimple_bb(), loop::inner, and basic_block_def::loop_father.
| tree stmt_vectype | ( | ) |
@verbatim Statement Analysis and Transformation for Vectorization
Copyright (C) 2003-2013 Free Software Foundation, Inc. Contributed by Dorit Naishlos dorit@il.ibm.com and Ira Rosen irar@il.ibm.com
This file is part of GCC.
GCC is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.
GCC is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with GCC; see the file COPYING3. If not see http://www.gnu.org/licenses/.
For lang_hooks.types.type_for_mode.
Return the vectorized type for the given statement.
| bool supportable_narrowing_operation | ( | enum tree_code | code, |
| tree | vectype_out, | ||
| tree | vectype_in, | ||
| enum tree_code * | code1, | ||
| int * | multi_step_cvt, | ||
| vec< tree > * | interm_types | ||
| ) |
Function supportable_narrowing_operation Check whether an operation represented by the code CODE is a narrowing operation that is supported by the target platform in vector form (i.e., when operating on arguments of type VECTYPE_IN and producing a result of type VECTYPE_OUT). Narrowing operations we currently support are NOP (CONVERT) and FIX_TRUNC. This function checks if these operations are supported by the target platform directly via vector tree-codes. Output: - CODE1 is the code of a vector operation to be used when vectorizing the operation, if available. - MULTI_STEP_CVT determines the number of required intermediate steps in case of multi-step conversion (like int->short->char - in that case MULTI_STEP_CVT will be 1). - INTERM_TYPES contains the intermediate type required to perform the narrowing operation (short in the above example).
References insn_data, insn_data_d::operand, optab_default, optab_for_tree_code(), optab_handler(), lang_hooks_for_types::type_for_mode, lang_hooks::types, and unknown_optab.
Referenced by vectorizable_conversion().
| bool supportable_widening_operation | ( | enum tree_code | code, |
| gimple | stmt, | ||
| tree | vectype_out, | ||
| tree | vectype_in, | ||
| enum tree_code * | code1, | ||
| enum tree_code * | code2, | ||
| int * | multi_step_cvt, | ||
| vec< tree > * | interm_types | ||
| ) |
Function supportable_widening_operation Check whether an operation represented by the code CODE is a widening operation that is supported by the target platform in vector form (i.e., when operating on arguments of type VECTYPE_IN producing a result of type VECTYPE_OUT). Widening operations we currently support are NOP (CONVERT), FLOAT and WIDEN_MULT. This function checks if these operations are supported by the target platform either directly (via vector tree-codes), or via target builtins. Output: - CODE1 and CODE2 are codes of vector operations to be used when vectorizing the operation, if available. - MULTI_STEP_CVT determines the number of required intermediate steps in case of multi-step conversion (like char->short->int - in that case MULTI_STEP_CVT will be 1). - INTERM_TYPES contains the intermediate type required to perform the widening operation (short in the above example).
References insn_data, nested_in_vect_loop_p(), insn_data_d::operand, optab_default, optab_for_tree_code(), optab_handler(), supportable_widening_operation(), lang_hooks_for_types::type_for_mode, lang_hooks::types, vect_used_by_reduction, and vinfo_for_stmt().
Referenced by supportable_widening_operation(), vect_recog_widen_mult_pattern(), vect_recog_widen_shift_pattern(), and vectorizable_conversion().
| bool vect_analyze_stmt | ( | ) |
Make sure the statement is vectorizable.
References dump_enabled_p(), dump_generic_expr(), dump_gimple_stmt(), dump_printf(), dump_printf_loc(), get_vectype_for_scalar_type(), gimple_expr_type(), gimple_get_lhs(), gimple_has_volatile_ops(), gsi_end_p(), gsi_next(), gsi_stmt(), is_pattern_stmt_p(), reduc_vec_info_type, si, vect_constant_def, vect_external_def, vect_induction_def, vect_internal_def, vect_location, vect_nested_cycle, vect_reduction_def, vect_unknown_def_type, vect_unused_in_scope, vect_used_in_outer, vect_used_in_outer_by_reduction, vectorizable_assignment(), vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_live_operation(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), vectorizable_shift(), vectorizable_store(), and vinfo_for_stmt().
Referenced by vect_analyze_loop_operations(), and vect_slp_analyze_node_operations().
|
static |
Function vect_cost_group_size For grouped load or store, return the group_size only if it is the first load or store of a group, else return 1. This ensures that group size is only returned once per group.
References first_stmt().
Referenced by vect_model_load_cost(), and vect_model_store_cost().
|
static |
Create vectorized demotion statements for vector operands from VEC_OPRNDS. For multi-step conversions store the resulting vectors and call the function recursively.
References gimple_assign_set_lhs(), gimple_build_assign_with_ops(), make_ssa_name(), vect_finish_stmt_generation(), and vinfo_for_stmt().
Referenced by vectorizable_conversion().
|
static |
Create vectorized promotion statements for vector operands from VEC_OPRNDS0 and VEC_OPRNDS1 (for binary operations). For multi-step conversions store the resulting vectors and call the function recursively.
References binary_op, gimple_assign_lhs(), gimple_call_lhs(), is_gimple_call(), vect_gen_widened_results_half(), and vNULL.
Referenced by vectorizable_conversion().
| void vect_finish_stmt_generation | ( | gimple | stmt, |
| gimple | vec_stmt, | ||
| gimple_stmt_iterator * | gsi | ||
| ) |
Function vect_finish_stmt_generation. Insert a new stmt.
References copy_ssa_name(), dump_enabled_p(), dump_gimple_stmt(), dump_printf_loc(), gimple_assign_lhs(), gimple_call_flags(), gimple_has_mem_ops(), gimple_location(), gimple_set_location(), gimple_set_vdef(), gimple_set_vuse(), gimple_vdef(), gimple_vuse(), gimple_vuse_op(), gsi_end_p(), gsi_insert_before(), GSI_SAME_STMT, gsi_stmt(), is_gimple_assign(), is_gimple_call(), is_gimple_reg(), new_stmt_vec_info(), set_vinfo_for_stmt(), vect_location, and vinfo_for_stmt().
Referenced by bump_vector_ptr(), permute_vec_elements(), read_vector_array(), vect_create_mask_and_perm(), vect_create_vectorized_demotion_stmts(), vect_gen_widened_results_half(), vect_init_vector_1(), vect_permute_load_chain(), vect_permute_store_chain(), vectorizable_assignment(), vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), vectorizable_shift(), vectorizable_store(), and write_vector_array().
| tree vect_gen_perm_mask | ( | ) |
Given a vector type VECTYPE and permutation SEL returns the VECTOR_CST mask that implements the permutation of the vector elements. If that is impossible to do, returns NULL.
References build_int_cst(), can_vec_perm_p(), get_vectype_for_scalar_type(), int_mode_for_mode(), lang_hooks_for_types::type_for_mode, and lang_hooks::types.
Referenced by perm_mask_for_reverse(), vect_permute_load_chain(), vect_permute_store_chain(), and vectorizable_load().
|
static |
Function vect_gen_widened_results_half Create a vector stmt whose code, type, number of arguments, and result variable are CODE, OP_TYPE, and VEC_DEST, and its arguments are VEC_OPRND0 and VEC_OPRND1. The new vector stmt is to be inserted at BSI. In the case that CODE is a CALL_EXPR, this means that a call to DECL needs to be created (DECL is a function-decl of a target-builtin). STMT is the original scalar stmt that we are vectorizing.
References binary_op, gimple_assign_set_lhs(), gimple_build_assign_with_ops(), gimple_build_call(), gimple_call_set_lhs(), make_ssa_name(), and vect_finish_stmt_generation().
Referenced by vect_create_vectorized_promotion_stmts().
| void vect_get_load_cost | ( | struct data_reference * | dr, |
| int | ncopies, | ||
| bool | add_realign_cost, | ||
| unsigned int * | inside_cost, | ||
| unsigned int * | prologue_cost, | ||
| stmt_vector_for_cost * | prologue_cost_vec, | ||
| stmt_vector_for_cost * | body_cost_vec, | ||
| bool | record_prologue_costs | ||
| ) |
Calculate cost of DR's memory access.
References dr_aligned, dr_explicit_realign, dr_explicit_realign_optimized, DR_STMT, dr_unaligned_supported, dr_unaligned_unsupported, dump_enabled_p(), dump_printf_loc(), record_stmt_cost(), data_reference::stmt, targetm, unaligned_load, vec_perm, vect_body, vect_location, vect_prologue, vect_supportable_dr_alignment(), vector_load, vector_stmt, and vinfo_for_stmt().
Referenced by vect_get_data_access_cost(), and vect_model_load_cost().
|
static |
Get vectorized definitions for loop-based vectorization. For the first operand we call vect_get_vec_def_for_operand() (with OPRND containing scalar operand), and for the rest we get a copy with vect_get_vec_def_for_stmt_copy() using the previous vector definition (stored in OPRND). See vect_get_vec_def_for_stmt_copy() for details. The vectors are collected into VEC_OPRNDS.
References vect_get_vec_def_for_operand(), and vect_get_vec_def_for_stmt_copy().
Referenced by vectorizable_conversion().
| void vect_get_store_cost | ( | struct data_reference * | dr, |
| int | ncopies, | ||
| unsigned int * | inside_cost, | ||
| stmt_vector_for_cost * | body_cost_vec | ||
| ) |
Calculate cost of DR's memory access.
References dr_aligned, DR_STMT, dr_unaligned_supported, dr_unaligned_unsupported, dump_enabled_p(), dump_printf_loc(), record_stmt_cost(), data_reference::stmt, unaligned_store, vect_body, vect_location, vect_supportable_dr_alignment(), vector_store, and vinfo_for_stmt().
Referenced by vect_get_data_access_cost(), and vect_model_store_cost().
| tree vect_get_vec_def_for_operand | ( | ) |
Function vect_get_vec_def_for_operand. OP is an operand in STMT. This function returns a (vector) def that will be used in the vectorized stmt for STMT. In the case that OP is an SSA_NAME which is defined in the loop, then STMT_VINFO_VEC_STMT of the defining stmt holds the relevant def. In case OP is an invariant or constant, a new stmt that creates a vector def needs to be introduced.
References dump_enabled_p(), dump_generic_expr(), dump_gimple_stmt(), dump_printf(), dump_printf_loc(), get_initial_def_for_reduction(), get_vectype_for_scalar_type(), gimple_assign_lhs(), gimple_bb(), gimple_call_lhs(), gimple_get_lhs(), is_gimple_call(), loop_preheader_edge(), vect_constant_def, vect_double_reduction_def, vect_external_def, vect_induction_def, vect_init_vector(), vect_internal_def, vect_is_simple_use(), vect_location, vect_nested_cycle, vect_reduction_def, and vinfo_for_stmt().
Referenced by get_initial_def_for_induction(), get_initial_def_for_reduction(), vect_create_epilog_for_reduction(), vect_get_loop_based_defs(), vect_get_vec_defs(), vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), and vectorizable_store().
| tree vect_get_vec_def_for_stmt_copy | ( | ) |
Function vect_get_vec_def_for_stmt_copy
Return a vector-def for an operand. This function is used when the
vectorized stmt to be created (by the caller to this function) is a "copy"
created in case the vectorized result cannot fit in one vector, and several
copies of the vector-stmt are required. In this case the vector-def is
retrieved from the vector stmt recorded in the STMT_VINFO_RELATED_STMT field
of the stmt that defines VEC_OPRND.
DT is the type of the vector def VEC_OPRND.
Context:
In case the vectorization factor (VF) is bigger than the number
of elements that can fit in a vectype (nunits), we have to generate
more than one vector stmt to vectorize the scalar stmt. This situation
arises when there are multiple data-types operated upon in the loop; the
smallest data-type determines the VF, and as a result, when vectorizing
stmts operating on wider types we need to create 'VF/nunits' "copies" of the
vector stmt (each computing a vector of 'nunits' results, and together
computing 'VF' results in each iteration). This function is called when
vectorizing such a stmt (e.g. vectorizing S2 in the illustration below, in
which VF=16 and nunits=4, so the number of copies required is 4):
scalar stmt: vectorized into: STMT_VINFO_RELATED_STMT
S1: x = load VS1.0: vx.0 = memref0 VS1.1
VS1.1: vx.1 = memref1 VS1.2
VS1.2: vx.2 = memref2 VS1.3
VS1.3: vx.3 = memref3
S2: z = x + ... VSnew.0: vz0 = vx.0 + ... VSnew.1
VSnew.1: vz1 = vx.1 + ... VSnew.2
VSnew.2: vz2 = vx.2 + ... VSnew.3
VSnew.3: vz3 = vx.3 + ...
The vectorization of S1 is explained in vectorizable_load.
The vectorization of S2:
To create the first vector-stmt out of the 4 copies - VSnew.0 -
the function 'vect_get_vec_def_for_operand' is called to
get the relevant vector-def for each operand of S2. For operand x it
returns the vector-def 'vx.0'.
To create the remaining copies of the vector-stmt (VSnew.j), this
function is called to get the relevant vector-def for each operand. It is
obtained from the respective VS1.j stmt, which is recorded in the
STMT_VINFO_RELATED_STMT field of the stmt that defines VEC_OPRND.
For example, to obtain the vector-def 'vx.1' in order to create the
vector stmt 'VSnew.1', this function is called with VEC_OPRND='vx.0'.
Given 'vx0' we obtain the stmt that defines it ('VS1.0'); from the
STMT_VINFO_RELATED_STMT field of 'VS1.0' we obtain the next copy - 'VS1.1',
and return its def ('vx.1').
Overall, to create the above sequence this function will be called 3 times:
vx.1 = vect_get_vec_def_for_stmt_copy (dt, vx.0);
vx.2 = vect_get_vec_def_for_stmt_copy (dt, vx.1);
vx.3 = vect_get_vec_def_for_stmt_copy (dt, vx.2);
References gimple_get_lhs(), vect_constant_def, vect_external_def, and vinfo_for_stmt().
Referenced by vect_create_epilog_for_reduction(), vect_get_loop_based_defs(), vect_get_vec_defs_for_stmt_copy(), vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), and vectorizable_store().
| void vect_get_vec_defs | ( | tree | op0, |
| tree | op1, | ||
| gimple | stmt, | ||
| vec< tree > * | vec_oprnds0, | ||
| vec< tree > * | vec_oprnds1, | ||
| slp_tree | slp_node, | ||
| int | reduc_index | ||
| ) |
Get vectorized definitions for OP0 and OP1. REDUC_INDEX is the index of reduction operand in case of reduction, and -1 otherwise.
References vect_get_slp_defs(), and vect_get_vec_def_for_operand().
Referenced by vect_create_epilog_for_reduction(), vectorizable_assignment(), vectorizable_conversion(), vectorizable_operation(), vectorizable_reduction(), vectorizable_shift(), and vectorizable_store().
|
static |
Get vectorized definitions for the operands to create a copy of an original stmt. See vect_get_vec_def_for_stmt_copy () for details.
References vect_get_vec_def_for_stmt_copy().
Referenced by vectorizable_assignment(), vectorizable_conversion(), vectorizable_operation(), and vectorizable_shift().
| tree vect_init_vector | ( | ) |
Function vect_init_vector. Insert a new stmt (INIT_STMT) that initializes a new variable of type TYPE with the value VAL. If TYPE is a vector type and VAL does not have vector type a vector with all elements equal to VAL is created first. Place the initialization at BSI if it is not NULL. Otherwise, place the initialization at the loop preheader. Return the DEF of INIT_STMT. It will be used in the vectorization of STMT.
References build_vector_from_val(), gimple_assign_lhs(), gimple_assign_set_lhs(), gimple_build_assign_with_ops(), make_ssa_name(), types_compatible_p(), vect_get_new_vect_var(), vect_init_vector_1(), and vect_simple_var.
Referenced by get_initial_def_for_induction(), vect_create_epilog_for_reduction(), vect_get_constant_vectors(), vect_get_vec_def_for_operand(), vectorizable_load(), and vectorizable_shift().
|
static |
Insert the new stmt NEW_STMT at *GSI or at the appropriate place in the loop preheader for the vectorized stmt STMT.
References dump_enabled_p(), dump_gimple_stmt(), dump_printf_loc(), gsi_after_labels(), gsi_insert_before(), gsi_insert_on_edge_immediate(), GSI_SAME_STMT, loop::inner, loop_preheader_edge(), nested_in_vect_loop_p(), vect_finish_stmt_generation(), vect_location, and vinfo_for_stmt().
Referenced by vect_init_vector(), and vectorizable_call().
|
static |
Function vect_is_simple_cond. Input: LOOP - the loop that is being vectorized. COND - Condition that is checked for simple use. Output: *COMP_VECTYPE - the vector type for the comparison. Returns whether a COND can be vectorized. Checks whether condition operands are supportable using vec_is_simple_use.
References vect_is_simple_use_1().
Referenced by vectorizable_condition().
| bool vect_is_simple_use | ( | tree | operand, |
| gimple | stmt, | ||
| loop_vec_info | loop_vinfo, | ||
| bb_vec_info | bb_vinfo, | ||
| gimple * | def_stmt, | ||
| tree * | def, | ||
| enum vect_def_type * | dt | ||
| ) |
Function vect_is_simple_use. Input: LOOP_VINFO - the vect info of the loop that is being vectorized. BB_VINFO - the vect info of the basic block that is being vectorized. OPERAND - operand of STMT in the loop or bb. DEF - the defining stmt in case OPERAND is an SSA_NAME. Returns whether a stmt with OPERAND can be vectorized. For loops, supportable operands are constants, loop invariants, and operands that are defined by the current iteration of the loop. Unsupportable operands are those that are defined by a previous iteration of the loop (as is the case in reduction/induction computations). For basic blocks, supportable operands are constants and bb invariants. For now, operands defined outside the basic block are not supported.
References dump_enabled_p(), dump_generic_expr(), dump_gimple_stmt(), dump_printf_loc(), flow_bb_inside_loop_p(), gimple_assign_lhs(), gimple_bb(), gimple_call_lhs(), gimple_nop_p(), gimple_phi_result(), is_gimple_min_invariant(), vect_constant_def, vect_double_reduction_def, vect_external_def, vect_location, vect_unknown_def_type, and vinfo_for_stmt().
Referenced by check_bool_pattern(), process_use(), type_conversion_p(), vect_analyze_slp_cost_1(), vect_get_and_check_slp_defs(), vect_get_vec_def_for_operand(), vect_is_simple_use_1(), vect_recog_rotate_pattern(), vect_recog_vector_vector_shift_pattern(), vectorizable_condition(), vectorizable_conversion(), vectorizable_live_operation(), vectorizable_operation(), vectorizable_reduction(), and vectorizable_store().
| bool vect_is_simple_use_1 | ( | tree | operand, |
| gimple | stmt, | ||
| loop_vec_info | loop_vinfo, | ||
| bb_vec_info | bb_vinfo, | ||
| gimple * | def_stmt, | ||
| tree * | def, | ||
| enum vect_def_type * | dt, | ||
| tree * | vectype | ||
| ) |
Function vect_is_simple_use_1. Same as vect_is_simple_use_1 but also determines the vector operand type of OPERAND and stores it to *VECTYPE. If the definition of OPERAND is vect_uninitialized_def, vect_constant_def or vect_external_def *VECTYPE will be set to NULL_TREE and the caller is responsible to compute the best suited vector type for the scalar operand.
References vect_constant_def, vect_double_reduction_def, vect_external_def, vect_induction_def, vect_internal_def, vect_is_simple_use(), vect_nested_cycle, vect_reduction_def, vect_uninitialized_def, and vinfo_for_stmt().
Referenced by vect_is_simple_cond(), vectorizable_assignment(), vectorizable_call(), vectorizable_conversion(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), and vectorizable_shift().
|
static |
Utility functions used by vect_mark_stmts_to_be_vectorized.
Function vect_mark_relevant. Mark STMT as "relevant for vectorization" and add it to WORKLIST.
References dump_enabled_p(), dump_printf_loc(), flow_bb_inside_loop_p(), gimple_assign_lhs(), gimple_call_lhs(), is_gimple_assign(), is_gimple_debug(), vect_location, and vinfo_for_stmt().
Referenced by process_use(), and vect_mark_stmts_to_be_vectorized().
| bool vect_mark_stmts_to_be_vectorized | ( | ) |
Function vect_mark_stmts_to_be_vectorized.
Not all stmts in the loop need to be vectorized. For example:
for i...
for j...
1. T0 = i + j
2. T1 = a[T0]
3. j = j + 1
Stmt 1 and 3 do not need to be vectorized, because loop control and
addressing of vectorized data-refs are handled differently.
This pass detects such stmts.
References dump_enabled_p(), dump_gimple_stmt(), dump_printf_loc(), gimple_assign_rhs1(), gimple_assign_rhs_code(), gimple_call_arg(), gimple_call_num_args(), gimple_num_ops(), gimple_op(), gsi_end_p(), gsi_next(), gsi_start_bb(), gsi_start_phis(), gsi_stmt(), is_gimple_assign(), is_gimple_call(), is_pattern_stmt_p(), loop::num_nodes, process_use(), si, vect_check_gather(), vect_double_reduction_def, vect_location, vect_mark_relevant(), vect_nested_cycle, vect_reduction_def, vect_stmt_relevant_p(), vect_unused_in_scope, vect_used_by_reduction, vect_used_in_outer, vect_used_in_outer_by_reduction, vinfo_for_stmt(), and worklist.
Referenced by vect_analyze_loop_2().
| void vect_model_load_cost | ( | stmt_vec_info | stmt_info, |
| int | ncopies, | ||
| bool | load_lanes_p, | ||
| slp_tree | slp_node, | ||
| stmt_vector_for_cost * | prologue_cost_vec, | ||
| stmt_vector_for_cost * | body_cost_vec | ||
| ) |
Function vect_model_load_cost Models cost for loads. In the case of grouped accesses, the last access has the overhead of the grouped access attributed to it. Since unaligned accesses are supported for loads, we also account for the costs of the access scheme chosen.
References dump_enabled_p(), dump_printf_loc(), exact_log2(), first_stmt(), record_stmt_cost(), scalar_load, vec_construct, vec_perm, vect_body, vect_cost_group_size(), vect_get_load_cost(), vect_location, and vinfo_for_stmt().
Referenced by vect_analyze_slp_cost_1(), and vectorizable_load().
|
static |
Model cost for type demotion and promotion operations. PWR is normally zero for single-step promotions and demotions. It will be one if two-step promotion/demotion is required, and so on. Each additional step doubles the number of instructions required.
References add_stmt_cost(), dump_enabled_p(), dump_printf_loc(), type_promotion_vec_info_type, vec_promote_demote, vect_body, vect_constant_def, vect_external_def, vect_location, vect_pow2(), vect_prologue, and vector_stmt.
Referenced by vectorizable_conversion().
| void vect_model_simple_cost | ( | stmt_vec_info | stmt_info, |
| int | ncopies, | ||
| enum vect_def_type * | dt, | ||
| stmt_vector_for_cost * | prologue_cost_vec, | ||
| stmt_vector_for_cost * | body_cost_vec | ||
| ) |
Function vect_model_simple_cost. Models cost for simple operations, i.e. those that only emit ncopies of a single op. Right now, this does not account for multiple insns that could be generated for the single vector op. We will handle that shortly.
References dump_enabled_p(), dump_printf_loc(), record_stmt_cost(), vect_body, vect_constant_def, vect_external_def, vect_location, vect_prologue, and vector_stmt.
Referenced by vectorizable_assignment(), vectorizable_call(), vectorizable_conversion(), vectorizable_operation(), and vectorizable_shift().
| void vect_model_store_cost | ( | stmt_vec_info | stmt_info, |
| int | ncopies, | ||
| bool | store_lanes_p, | ||
| enum vect_def_type | dt, | ||
| slp_tree | slp_node, | ||
| stmt_vector_for_cost * | prologue_cost_vec, | ||
| stmt_vector_for_cost * | body_cost_vec | ||
| ) |
Function vect_model_store_cost Models cost for stores. In the case of grouped accesses, one access has the overhead of the grouped access attributed to it.
References dump_enabled_p(), dump_printf_loc(), exact_log2(), first_stmt(), record_stmt_cost(), scalar_to_vec, vec_perm, vect_body, vect_constant_def, vect_cost_group_size(), vect_external_def, vect_get_store_cost(), vect_location, vect_prologue, and vinfo_for_stmt().
Referenced by vect_analyze_slp_cost_1(), and vectorizable_store().
| void vect_remove_stores | ( | ) |
Remove a group of stores (for SLP or interleaving), free their stmt_vec_info.
References first_stmt(), free_stmt_vec_info(), gsi_for_stmt(), gsi_remove(), is_pattern_stmt_p(), loop::next, release_defs(), unlink_stmt_vdef(), and vinfo_for_stmt().
Referenced by vect_transform_loop().
|
static |
Function vect_stmt_relevant_p. Return true if STMT in loop that is represented by LOOP_VINFO is "relevant for vectorization". A stmt is considered "relevant for vectorization" if: - it has uses outside the loop. - it has vdefs (it alters memory). - control stmts in the loop (except for the exit condition). CHECKME: what other side effects would the vectorizer allow?
References dump_enabled_p(), dump_printf_loc(), flow_bb_inside_loop_p(), gimple_bb(), gimple_vdef(), is_ctrl_stmt(), is_gimple_debug(), loop_exit_ctrl_vec_info_type, single_exit(), vect_location, vect_unused_in_scope, vect_used_in_scope, and vinfo_for_stmt().
Referenced by vect_mark_stmts_to_be_vectorized().
| bool vect_supportable_shift | ( | ) |
Return TRUE if CODE (a shift operation) is supported for SCALAR_TYPE either as shift by a scalar or by a vector.
References get_vectype_for_scalar_type(), optab_for_tree_code(), optab_handler(), optab_scalar, and optab_vector.
Referenced by vect_operation_fits_smaller_type().
| bool vect_transform_stmt | ( | gimple | stmt, |
| gimple_stmt_iterator * | gsi, | ||
| bool * | grouped_store, | ||
| slp_tree | slp_node, | ||
| slp_instance | slp_node_instance | ||
| ) |
Function vect_transform_stmt. Create a vectorized stmt to replace STMT, and insert it at BSI.
References assignment_vec_info_type, call_vec_info_type, condition_vec_info_type, dump_enabled_p(), dump_printf_loc(), flow_bb_inside_loop_p(), gimple_assign_lhs(), gsi_stmt(), induc_vec_info_type, loop::inner, load_vec_info_type, nested_in_vect_loop_p(), op_vec_info_type, reduc_vec_info_type, shift_vec_info_type, store_vec_info_type, type_conversion_vec_info_type, type_demotion_vec_info_type, type_promotion_vec_info_type, vect_location, vect_used_in_outer, vect_used_in_outer_by_reduction, vectorizable_assignment(), vectorizable_call(), vectorizable_condition(), vectorizable_conversion(), vectorizable_induction(), vectorizable_live_operation(), vectorizable_load(), vectorizable_operation(), vectorizable_reduction(), vectorizable_shift(), vectorizable_store(), and vinfo_for_stmt().
Referenced by vect_schedule_slp_instance(), and vect_transform_loop().
|
static |
Function vectorizable_assignment. Check if STMT performs an assignment (copy) that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References assignment_vec_info_type, dump_enabled_p(), dump_printf_loc(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), gimple_assign_single_p(), is_gimple_assign(), make_ssa_name(), vect_create_destination_var(), vect_finish_stmt_generation(), vect_get_vec_defs(), vect_get_vec_defs_for_stmt_copy(), vect_internal_def, vect_is_simple_use_1(), vect_location, vect_model_simple_cost(), vect_unknown_def_type, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
|
static |
Function vectorizable_call. Check if STMT performs a function call that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References build_int_cst(), build_zero_cst(), call_vec_info_type, dump_enabled_p(), dump_generic_expr(), dump_printf_loc(), get_same_sized_vectype(), gimple_assign_lhs(), gimple_assign_set_lhs(), gimple_build_call_vec(), gimple_call_arg(), gimple_call_internal_fn(), gimple_call_internal_p(), gimple_call_lhs(), gimple_call_num_args(), gimple_call_set_lhs(), gimple_purge_dead_eh_edges(), gimple_vuse(), gsi_replace(), is_gimple_call(), is_pattern_stmt_p(), make_ssa_name(), maybe_clean_or_replace_eh_stmt(), set_vinfo_for_stmt(), loop::simduid, stmt_can_throw_internal(), type(), types_compatible_p(), vect_create_destination_var(), vect_finish_stmt_generation(), vect_get_new_vect_var(), vect_get_slp_defs(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_init_vector_1(), vect_internal_def, vect_is_simple_use_1(), vect_location, vect_model_simple_cost(), vect_simple_var, vect_unknown_def_type, vectorizable_function(), vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
| bool vectorizable_condition | ( | gimple | stmt, |
| gimple_stmt_iterator * | gsi, | ||
| gimple * | vec_stmt, | ||
| tree | reduc_def, | ||
| int | reduc_index, | ||
| slp_tree | slp_node | ||
| ) |
vectorizable_condition. Check if STMT is conditional modify expression that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt using VEC_COND_EXPR to replace it, put it in VEC_STMT, and insert it at GSI. When STMT is vectorized as nested cycle, REDUC_DEF is the vector variable to be used at REDUC_INDEX (in then clause if REDUC_INDEX is 1, and in else caluse if it is 2). Return FALSE if not a vectorizable STMT, TRUE otherwise.
References build_nonstandard_integer_type(), condition_vec_info_type, dump_enabled_p(), dump_printf_loc(), expand_vec_cond_expr_p(), get_same_sized_vectype(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_rhs2(), gimple_assign_rhs3(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), is_gimple_assign(), make_ssa_name(), vect_create_destination_var(), vect_finish_stmt_generation(), vect_get_slp_defs(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_internal_def, vect_is_simple_cond(), vect_is_simple_use(), vect_location, vect_nested_cycle, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), vect_transform_stmt(), and vectorizable_reduction().
|
static |
Check if STMT performs a conversion operation, that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at GSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References binary_op, build_nonstandard_integer_type(), dump_enabled_p(), dump_generic_expr(), dump_printf_loc(), get_same_sized_vectype(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_rhs2(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), gimple_build_assign_with_ops(), gimple_build_call(), gimple_call_set_lhs(), is_gimple_assign(), make_ssa_name(), supportable_convert_operation(), supportable_narrowing_operation(), supportable_widening_operation(), type_conversion_vec_info_type, type_demotion_vec_info_type, type_promotion_vec_info_type, unary_op, _slp_tree::vec_stmts_size, vect_create_destination_var(), vect_create_vectorized_demotion_stmts(), vect_create_vectorized_promotion_stmts(), vect_finish_stmt_generation(), vect_get_loop_based_defs(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_get_vec_defs(), vect_get_vec_defs_for_stmt_copy(), vect_internal_def, vect_is_simple_use(), vect_is_simple_use_1(), vect_location, vect_model_promotion_demotion_cost(), vect_model_simple_cost(), vect_pow2(), vect_unknown_def_type, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
| tree vectorizable_function | ( | ) |
Checks if CALL can be vectorized in type VECTYPE. Returns a function declaration if the target has a vectorized version of the function, or NULL_TREE if the function cannot be vectorized.
References gimple_call_flags(), gimple_call_fndecl(), and targetm.
Referenced by vect_recog_pow_pattern(), and vectorizable_call().
|
static |
vectorizable_load. Check if STMT reads a non scalar data-ref (array/pointer/structure) that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References add_phi_arg(), alias_sets_conflict_p(), aligned_access_p(), build_aligned_type(), build_array_type_nelts(), build_constructor(), build_int_cst(), build_real(), build_vector_from_val(), bump_vector_ptr(), copy_ssa_name(), create_array_ref(), create_iv(), create_vector_array(), dr_aligned, DR_BASE_ADDRESS, dr_explicit_realign, dr_explicit_realign_optimized, DR_INIT, DR_OFFSET, DR_REF, DR_STEP, dr_unaligned_supported, dump_enabled_p(), dump_printf_loc(), first_stmt(), force_gimple_operand(), force_gimple_operand_gsi(), get_alias_set(), get_ptr_info(), gimple_assign_lhs(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), gimple_bb(), gimple_build_assign_with_ops(), gimple_build_call(), gimple_build_call_internal(), gimple_call_set_lhs(), gimple_set_vdef(), gimple_set_vuse(), gimple_vdef(), gimple_vuse(), gsi_insert_seq_on_edge_immediate(), gsi_next(), GSI_SAME_STMT, gsi_stmt(), HOST_WIDE_INT, int_const_binop(), integer_zerop(), is_gimple_assign(), is_gimple_min_invariant(), load_vec_info_type, loop_latch_edge(), loop_preheader_edge(), make_ssa_name(), nested_in_vect_loop_p(), new_stmt_vec_info(), offset, optab_handler(), perm_mask_for_reverse(), permute_vec_elements(), read_vector_array(), real_from_target(), reference_alias_ptr_type(), set_ptr_info_alignment(), set_vinfo_for_stmt(), standard_iv_increment_position(), tcc_declaration, tree_int_cst_compare(), types_compatible_p(), unshare_expr(), useless_type_conversion_p(), vec_alloc(), vect_check_gather(), vect_create_data_ref_ptr(), vect_create_destination_var(), vect_finish_stmt_generation(), vect_gen_perm_mask(), vect_get_new_vect_var(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_grouped_load_supported(), vect_init_vector(), vect_internal_def, vect_is_simple_use_1(), vect_load_lanes_supported(), vect_location, vect_model_load_cost(), vect_record_grouped_load_vectors(), vect_setup_realignment(), vect_simple_var, vect_supportable_dr_alignment(), vect_transform_grouped_load(), vect_transform_slp_perm_load(), vect_unknown_def_type, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
|
static |
Function vectorizable_operation. Check if STMT performs a binary, unary or ternary operation that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References binary_op, can_mult_highpart_p(), dump_enabled_p(), dump_generic_expr(), dump_printf_loc(), get_same_sized_vectype(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_rhs2(), gimple_assign_rhs3(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), gimple_build_assign_with_ops(), is_gimple_assign(), make_ssa_name(), op_vec_info_type, optab_default, optab_for_tree_code(), optab_handler(), ternary_op, unary_op, vect_create_destination_var(), vect_finish_stmt_generation(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_get_vec_defs(), vect_get_vec_defs_for_stmt_copy(), vect_internal_def, vect_is_simple_use(), vect_is_simple_use_1(), vect_location, vect_min_worthwhile_factor(), vect_model_simple_cost(), vect_unknown_def_type, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
|
static |
Function vectorizable_shift. Check if STMT performs a shift operation that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References dump_enabled_p(), dump_printf_loc(), get_same_sized_vectype(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_rhs2(), gimple_assign_rhs_code(), gimple_assign_set_lhs(), gimple_build_assign_with_ops(), insn_data, is_gimple_assign(), make_ssa_name(), insn_data_d::operand, operand_equal_p(), optab_for_tree_code(), optab_handler(), optab_scalar, optab_vector, shift_vec_info_type, useless_type_conversion_p(), _slp_tree::vec_stmts_size, vect_constant_def, vect_create_destination_var(), vect_external_def, vect_finish_stmt_generation(), vect_get_vec_defs(), vect_get_vec_defs_for_stmt_copy(), vect_init_vector(), vect_internal_def, vect_is_simple_use_1(), vect_location, vect_min_worthwhile_factor(), vect_model_simple_cost(), vect_unknown_def_type, vinfo_for_stmt(), and vNULL.
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
|
static |
Function vectorizable_store. Check if STMT defines a non scalar data-ref (array/pointer/structure) that can be vectorized. If VEC_STMT is also passed, vectorize the STMT: create a vectorized stmt to replace it, put it in VEC_STMT, and insert it at BSI. Return FALSE if not a vectorizable STMT, TRUE otherwise.
Transform. *
References alias_sets_conflict_p(), aligned_access_p(), build_aligned_type(), build_array_type_nelts(), build_int_cst(), bump_vector_ptr(), create_array_ref(), create_vector_array(), dr_aligned, DR_BASE_ADDRESS, DR_INIT, DR_OFFSET, DR_REF, DR_STEP, dr_unaligned_supported, dump_enabled_p(), dump_printf_loc(), first_stmt(), get_alias_set(), get_ptr_info(), gimple_assign_lhs(), gimple_assign_rhs1(), gimple_assign_single_p(), gimple_build_call_internal(), gimple_call_set_lhs(), int_const_binop(), integer_zerop(), is_gimple_assign(), is_pattern_stmt_p(), nested_in_vect_loop_p(), optab_handler(), reference_alias_ptr_type(), set_ptr_info_alignment(), store_vec_info_type, tree_int_cst_compare(), unshare_expr(), useless_type_conversion_p(), vect_create_data_ref_ptr(), vect_finish_stmt_generation(), vect_get_vec_def_for_operand(), vect_get_vec_def_for_stmt_copy(), vect_get_vec_defs(), vect_grouped_store_supported(), vect_internal_def, vect_is_simple_use(), vect_location, vect_model_store_cost(), vect_permute_store_chain(), vect_store_lanes_supported(), vect_supportable_dr_alignment(), vinfo_for_stmt(), vNULL, and write_vector_array().
Referenced by vect_analyze_stmt(), and vect_transform_stmt().
|
static |
ARRAY is an array of vectors created by create_vector_array. Emit code to store SSA_NAME VECT in index N of the array. The store is part of the vectorization of STMT.
References build_int_cst(), and vect_finish_stmt_generation().
Referenced by vectorizable_store().
Variable Documentation
| unsigned int current_vector_size |
In tree-vect-stmts.c.
Referenced by get_vectype_for_scalar_type(), vect_analyze_loop(), and vect_slp_analyze_bb().
