## The Annals of Probability

### Reflection Groups, Generalized Schur Functions, and the Geometry of Majorization

#### Abstract

Let $G$ be a closed subgroup of the orthogonal group $O(n)$ acting on $R^n$. A real-valued function $f$ on $R^n$ is called $G$-monotone (decreasing) if $f(y) \geqq f(x)$ whenever $y \precsim x$, i.e., whenever $y \in C(x)$, where $C(x)$ is the convex hull of the $G$-orbit of $x$. When $G$ is the permutation group $\mathscr{P}_n$ the ordering $\sim$ is the majorization ordering of Schur, and the $\mathscr{P}_n$-monotone functions are the Schur-concave functions. This paper contains a geometrical study of the convex polytopes $C(x)$ and the ordering $\precsim$ when $G$ is any closed subgroup of $O(n)$ that is generated by reflections, which includes $\mathscr{P}_n$ as a special case. The classical results of Schur (1923), Ostrowski (1952), Rado (1952), and Hardy, Littlewood and Polya (1952) concerning majorization and Schur functions are generalized to reflection groups. It is shown that a smooth $G$-invariant function $f$ is $G$-monotone iff $(r'x)(r'\nabla f(x))\leqq 0$ for all $x \in R^n$ and all $r \in R^n$ such that the reflection across the hyperplane $\{z\mid r'z = 0\}$ is in $G$. Furthermore, it is shown that the convolution (relative to Lebesgue measure) of two nonnegative $G$-monotone functions is again $G$-monotone. The latter extends a theorem of Marshall and Olkin (1974) concerning $\mathscr{P}_n$, and has applications to probability inequalities arising in multivariate statistical analysis.

#### Article information

Source
Ann. Probab., Volume 5, Number 6 (1977), 829-860.

Dates
First available in Project Euclid: 19 April 2007

https://projecteuclid.org/euclid.aop/1176995655

Digital Object Identifier
doi:10.1214/aop/1176995655

Mathematical Reviews number (MathSciNet)
MR444864

Zentralblatt MATH identifier
0401.51006

JSTOR