Open Access
Translator Disclaimer
2021 A new framework for distance and kernel-based metrics in high dimensions
Shubhadeep Chakraborty, Xianyang Zhang
Author Affiliations +
Electron. J. Statist. 15(2): 5455-5522 (2021). DOI: 10.1214/21-EJS1889

Abstract

The paper presents new metrics to quantify and test for (i) the equality of distributions and (ii) the independence between two high-dimensional random vectors. We show that the energy distance based on the usual Euclidean distance cannot completely characterize the homogeneity of two high-dimensional distributions in the sense that it only detects the equality of means and the traces of covariance matrices in the high-dimensional setup. We propose a new class of metrics which inherits the desirable properties of the energy distance and maximum mean discrepancy/(generalized) distance covariance and the Hilbert-Schmidt Independence Criterion in the low-dimensional setting and is capable of detecting the homogeneity of/completely characterizing independence between the low-dimensional marginal distributions in the high dimensional setup. We further propose t-tests based on the new metrics to perform high-dimensional two-sample testing/independence testing and study their asymptotic behavior under both high dimension low sample size (HDLSS) and high dimension medium sample size (HDMSS) setups. The computational complexity of the t-tests only grows linearly with the dimension and thus is scalable to very high dimensional data. We demonstrate the superior power behavior of the proposed tests for homogeneity of distributions and independence via both simulated and real datasets.

Acknowledgments

We are grateful to the editor, the anonymous associate editor and two very careful reviewers for their constructive comments and suggestions which helped us improve the manuscript.

Citation

Download Citation

Shubhadeep Chakraborty. Xianyang Zhang. "A new framework for distance and kernel-based metrics in high dimensions." Electron. J. Statist. 15 (2) 5455 - 5522, 2021. https://doi.org/10.1214/21-EJS1889

Information

Received: 1 November 2020; Published: 2021
First available in Project Euclid: 15 December 2021

Digital Object Identifier: 10.1214/21-EJS1889

Keywords: distance covariance , energy distance , high dimensionality , Hilbert-Schmidt independence criterion , Independence test , maximum mean discrepancy , two sample test , U-statistic

JOURNAL ARTICLE
68 PAGES


SHARE
Vol.15 • No. 2 • 2021
Back to Top