Yang Yang Research Project

Yang Yang (UIC graduate student)

Title: Conditional Independent Tests with DeepBET

Abstract: Conditional independence (CI) serves as a fundamental cornerstone in statistics, machine learning, and artificial intelligence. This project focuses on the assessment of conditional independence between two random univariate variables, X and Y, given a set of high-dimensional confounding variables Z. The dimensionality of Z poses a challenge for many existing tests, leading to either inflated type-I errors or insufficient power in detecting alternatives.

To address this issue, we leverage the Deep Neural Network (DNN)’s ability to handle complex, high-dimensional data while circumventing the curse of dimensionality. We propose the utilization of a DNN model to estimate the conditional means of X and Y given Z using part of the data and obtain predicted errors using the other part of the data. We then apply novel binary expansion statistics as our test metrics to predicted errors for dependence detection. Furthermore, we implement the multiple split method to enhance power, utilizing the entirety of the sample while minimizing randomness. Our preliminary results show that the proposed method adeptly controls type I error control and exhibits a significant capacity to detect alternatives, making it a robust approach for testing conditional independence in the presence of high-dimensional confounding variables.