Translator Disclaimer
March 2021 Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery
David K. Lim, Naim U. Rashid, Joseph G. Ibrahim
Author Affiliations +
Ann. Appl. Stat. 15(1): 481-508 (March 2021). DOI: 10.1214/20-AOAS1407


Clustering is a form of unsupervised learning that aims to uncover latent groups within data based on similarity across a set of features. A common application of this in biomedical research is in delineating novel cancer subtypes from patient gene expression data, given a set of informative genes. However, it is typically unknown a priori what genes may be informative in discriminating between clusters and what the optimal number of clusters are. Few methods exist for performing unsupervised clustering of RNA-seq samples, and none currently adjust for between-sample global normalization factors, select cluster-discriminatory genes or account for potential confounding variables during clustering. To address these issues, we propose the feature selection and clustering of RNA-seq (FSCseq): a model-based clustering algorithm that utilizes a finite mixture of regression (FMR) model and the quadratic penalty method with a smoothly clipped absolute deviation (SCAD) penalty. The maximization is done by a penalized Classification EM algorithm, allowing us to include normalization factors and confounders in our modeling framework. Given the fitted model, our framework allows for subtype prediction in new patients via posterior probabilities of cluster membership, even in the presence of batch effects. Based on simulations and real data analysis, we show the advantages of our method relative to competing approaches.


Download Citation

David K. Lim. Naim U. Rashid. Joseph G. Ibrahim. "Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery." Ann. Appl. Stat. 15 (1) 481 - 508, March 2021.


Received: 1 February 2020; Revised: 1 August 2020; Published: March 2021
First available in Project Euclid: 18 March 2021

Digital Object Identifier: 10.1214/20-AOAS1407

Keywords: batch , clustering , confounders , RNA-Seq

Rights: Copyright © 2021 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.15 • No. 1 • March 2021
Back to Top