Detecting Relevant Interactions in High Dimensional Data Analysis

In high dimensional data, relevant interactions can be difficult to identify due to the extremely large number of possible interactions among variables. Conventional methods use a screening stage to vastly reduce the dimension of the variable space before examining the interaction effect. However, this type of screening relies on the assumption that the relevant variables of interactions also have significantly strong marginal effects, which may not be the case in many applications. Hence we propose a multi-stage screening method that considers interaction effects to ensure that variables with weak marginal effect can still be identified if their interactions are strong. The proposed screening method can achieve the same reduction of dimensionality in linear computational time using a random sampling scheme. Experimental studies on three simulated scenarios, a financial news dataset and a biostatistics dataset show that the proposed approach can effectively identify variables that are of weak marginal effect but relevant interaction effect