September 2023 Probabilistic learning of treatment trees in cancer
Tsung-Hung Yao, Zhenke Wu, Karthik Bharath, Jinju Li, Veerabhadran Baladandayuthapani
Author Affiliations +
Ann. Appl. Stat. 17(3): 1884-1908 (September 2023). DOI: 10.1214/22-AOAS1696


Accurate identification of synergistic treatment combinations and their underlying biological mechanisms is critical across many disease domains, especially cancer. In translational oncology research, preclinical systems, such as patient-derived xenografts (PDX), have emerged as a unique study design evaluating multiple treatments administered to samples from the same human tumor implanted into genetically identical mice. In this paper we propose a novel Bayesian probabilistic tree-based framework for PDX data to investigate the hierarchical relationships between treatments by inferring treatment cluster trees, referred to as treatment trees (Rx-tree). The framework motivates a new metric of mechanistic similarity between two or more treatments, accounting for inherent uncertainty in tree estimation; treatments with a high estimated similarity have potentially high mechanistic synergy. Building upon Dirichlet diffusion trees, we derive a closed-form marginal likelihood, encoding the tree structure, which facilitates computationally efficient posterior inference via a new two-stage algorithm. Simulation studies demonstrate superior performance of the proposed method in recovering the tree structure and treatment similarities. Our analyses of a recently collated PDX dataset produce treatment similarity estimates that show a high degree of concordance with known biological mechanisms across treatments in five different cancers. More importantly, we uncover new and potentially effective combination therapies that confer synergistic regulation of specific downstream biological pathways for future clinical investigations. Our accompanying code, data, and shiny application for visualization of results are available at:


Download Citation

Tsung-Hung Yao. Zhenke Wu. Karthik Bharath. Jinju Li. Veerabhadran Baladandayuthapani. "Probabilistic learning of treatment trees in cancer." Ann. Appl. Stat. 17 (3) 1884 - 1908, September 2023.


Received: 1 January 2022; Revised: 1 July 2022; Published: September 2023
First available in Project Euclid: 7 September 2023

MathSciNet: MR4637649
Digital Object Identifier: 10.1214/22-AOAS1696

Keywords: Approximate Bayesian Computation , Dirichlet diffusion trees , patient derived xenograft , Precision medicine , tree-based clustering

Rights: Copyright © 2023 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.17 • No. 3 • September 2023
Back to Top