## The Annals of Applied Probability

- Ann. Appl. Probab.
- Volume 27, Number 3 (2017), 1646-1677.

### The shape of the one-dimensional phylogenetic likelihood function

Vu Dinh and Frederick A. Matsen IV

#### Abstract

By fixing all parameters in a phylogenetic likelihood model except for one branch length, one obtains a one-dimensional likelihood function. In this work, we introduce a mathematical framework to characterize the shapes of such one-dimensional phylogenetic likelihood functions. This framework is based on analyses of algebraic structures on the space of all frequency patterns with respect to a polynomial representation of the likelihood functions. Using this framework, we provide conditions under which the one-dimensional phylogenetic likelihood functions are guaranteed to have at most one stationary point, and this point is the maximum likelihood branch length. These conditions are satisfied by common simple models including all binary models, the Jukes–Cantor model and the Felsenstein 1981 model.

We then prove that for the simplest model that does not satisfy our conditions, namely, the Kimura 2-parameter model, the one-dimensional likelihood functions may have multiple stationary points. As a proof of concept, we construct a nondegenerate example in which the phylogenetic likelihood function has two local maxima and a local minimum. To construct such examples, we derive a general method of constructing a tree and sequence data with a specified frequency pattern at the root. We then extend the result to prove that the space of all rescaled and translated one-dimensional phylogenetic likelihood functions under the Kimura 2-parameter model is dense in the space of all nonnegative continuous functions on $[0,\infty)$ with finite limits. These results indicate that one-dimensional likelihood functions under advanced evolutionary models can be more complex than it is typically assumed by phylogenetic inference algorithms; however, these complexities can be effectively captured by the Kimura 2-parameter model.

#### Article information

**Source**

Ann. Appl. Probab. Volume 27, Number 3 (2017), 1646-1677.

**Dates**

Received: July 2015

Revised: July 2016

First available in Project Euclid: 19 July 2017

**Permanent link to this document**

https://projecteuclid.org/euclid.aoap/1500451237

**Digital Object Identifier**

doi:10.1214/16-AAP1240

**Subjects**

Primary: 05C05: Trees 92B10: Taxonomy, cladistics, statistics

Secondary: 05C25: Graphs and abstract algebra (groups, rings, fields, etc.) [See also 20F65] 92D15: Problems related to evolution

**Keywords**

Evolutionary model molecular evolution phylogenetics likelihood model characteristic polynomial algebraic representation multimodality universal model

#### Citation

Dinh, Vu; Matsen IV, Frederick A. The shape of the one-dimensional phylogenetic likelihood function. Ann. Appl. Probab. 27 (2017), no. 3, 1646--1677. doi:10.1214/16-AAP1240. https://projecteuclid.org/euclid.aoap/1500451237