Translator Disclaimer
June 2017 The shape of the one-dimensional phylogenetic likelihood function
Vu Dinh, Frederick A. Matsen IV
Ann. Appl. Probab. 27(3): 1646-1677 (June 2017). DOI: 10.1214/16-AAP1240

Abstract

By fixing all parameters in a phylogenetic likelihood model except for one branch length, one obtains a one-dimensional likelihood function. In this work, we introduce a mathematical framework to characterize the shapes of such one-dimensional phylogenetic likelihood functions. This framework is based on analyses of algebraic structures on the space of all frequency patterns with respect to a polynomial representation of the likelihood functions. Using this framework, we provide conditions under which the one-dimensional phylogenetic likelihood functions are guaranteed to have at most one stationary point, and this point is the maximum likelihood branch length. These conditions are satisfied by common simple models including all binary models, the Jukes–Cantor model and the Felsenstein 1981 model.

We then prove that for the simplest model that does not satisfy our conditions, namely, the Kimura 2-parameter model, the one-dimensional likelihood functions may have multiple stationary points. As a proof of concept, we construct a nondegenerate example in which the phylogenetic likelihood function has two local maxima and a local minimum. To construct such examples, we derive a general method of constructing a tree and sequence data with a specified frequency pattern at the root. We then extend the result to prove that the space of all rescaled and translated one-dimensional phylogenetic likelihood functions under the Kimura 2-parameter model is dense in the space of all nonnegative continuous functions on $[0,\infty)$ with finite limits. These results indicate that one-dimensional likelihood functions under advanced evolutionary models can be more complex than it is typically assumed by phylogenetic inference algorithms; however, these complexities can be effectively captured by the Kimura 2-parameter model.

Citation

Download Citation

Vu Dinh. Frederick A. Matsen IV. "The shape of the one-dimensional phylogenetic likelihood function." Ann. Appl. Probab. 27 (3) 1646 - 1677, June 2017. https://doi.org/10.1214/16-AAP1240

Information

Received: 1 July 2015; Revised: 1 July 2016; Published: June 2017
First available in Project Euclid: 19 July 2017

zbMATH: 1370.05039
MathSciNet: MR3678481
Digital Object Identifier: 10.1214/16-AAP1240

Subjects:
Primary: 05C05, 92B10
Secondary: 05C25, 92D15

Rights: Copyright © 2017 Institute of Mathematical Statistics

JOURNAL ARTICLE
32 PAGES


SHARE
Vol.27 • No. 3 • June 2017
Back to Top