## The Annals of Probability

- Ann. Probab.
- Volume 5, Number 6 (1977), 829-860.

### Reflection Groups, Generalized Schur Functions, and the Geometry of Majorization

Morris L. Eaton and Michael D. Perlman

#### Abstract

Let $G$ be a closed subgroup of the orthogonal group $O(n)$ acting on $R^n$. A real-valued function $f$ on $R^n$ is called $G$-monotone (decreasing) if $f(y) \geqq f(x)$ whenever $y \precsim x$, i.e., whenever $y \in C(x)$, where $C(x)$ is the convex hull of the $G$-orbit of $x$. When $G$ is the permutation group $\mathscr{P}_n$ the ordering $\sim$ is the majorization ordering of Schur, and the $\mathscr{P}_n$-monotone functions are the Schur-concave functions. This paper contains a geometrical study of the convex polytopes $C(x)$ and the ordering $\precsim$ when $G$ is any closed subgroup of $O(n)$ that is generated by reflections, which includes $\mathscr{P}_n$ as a special case. The classical results of Schur (1923), Ostrowski (1952), Rado (1952), and Hardy, Littlewood and Polya (1952) concerning majorization and Schur functions are generalized to reflection groups. It is shown that a smooth $G$-invariant function $f$ is $G$-monotone iff $(r'x)(r'\nabla f(x))\leqq 0$ for all $x \in R^n$ and all $r \in R^n$ such that the reflection across the hyperplane $\{z\mid r'z = 0\}$ is in $G$. Furthermore, it is shown that the convolution (relative to Lebesgue measure) of two nonnegative $G$-monotone functions is again $G$-monotone. The latter extends a theorem of Marshall and Olkin (1974) concerning $\mathscr{P}_n$, and has applications to probability inequalities arising in multivariate statistical analysis.

#### Article information

**Source**

Ann. Probab., Volume 5, Number 6 (1977), 829-860.

**Dates**

First available in Project Euclid: 19 April 2007

**Permanent link to this document**

https://projecteuclid.org/euclid.aop/1176995655

**Digital Object Identifier**

doi:10.1214/aop/1176995655

**Mathematical Reviews number (MathSciNet)**

MR444864

**Zentralblatt MATH identifier**

0401.51006

**JSTOR**

links.jstor.org

**Subjects**

Primary: 26A84

Secondary: 26A86 50B35 52A25 62H99: None of the above, but in this section

**Keywords**

Orthogonal transformations groups generated by reflections Coxeter groups roots fundamental region $G$-orbit convex hull convex polytope convex polyhedral cone extreme point extreme ray edge $G$-monotone majorization Schur-concave function convolution

#### Citation

Eaton, Morris L.; Perlman, Michael D. Reflection Groups, Generalized Schur Functions, and the Geometry of Majorization. Ann. Probab. 5 (1977), no. 6, 829--860. doi:10.1214/aop/1176995655. https://projecteuclid.org/euclid.aop/1176995655