## Journal of the Mathematical Society of Japan

- J. Math. Soc. Japan
- Volume 54, Number 1 (2002), 215-253.

### The Penrose transform for $Sp(n,R)$ and singular unitary representations

#### Abstract

We give a general definition of the Radon-Penrose transform for a Zuckerman-Vogan derived functor module of a reductive Lie group $G$, which maps from the Dolbeault cohomology group over a pseudo-Kähler homogeneous manifold into the space of smooth sections of a vector bundle over a Riemannian symmetric space. Furthermore, we formulate a functorial property between two Penrose transforms in the context of the Kobayashi theory of discretely decomposable restrictions of unitary representations.

Based on this general theory, we study the Penrose transform for a family of singular unitary representations of $Sp(n,R)$ in details. We prove that the image of the Penrose transform is exactly the space of global holomorphic solutions of the system of partial differential equations of minor determinant type of odd degree over the bounded symmetric domain of type CI, which is biholomorphic to the Siegel upper half space. This system might be regarded as a generalization of the Gauss-Aomoto-Gelfand hypergeometric differential equations to higher order. We also find a new phenomenon that the kernel of the Penrose transform is non-zero, which we determine explicitly by means of representation theory.

#### Article information

**Source**

J. Math. Soc. Japan, Volume 54, Number 1 (2002), 215-253.

**Dates**

First available in Project Euclid: 5 October 2007

**Permanent link to this document**

https://projecteuclid.org/euclid.jmsj/1191593961

**Digital Object Identifier**

doi:10.2969/jmsj/1191593961

**Mathematical Reviews number (MathSciNet)**

MR1864934

**Zentralblatt MATH identifier**

1020.32015

**Subjects**

Primary: 22E46: Semisimple Lie groups and their representations

Secondary: 32L10: Sheaves and cohomology of sections of holomorphic vector bundles, general results [See also 14F05, 18F20, 55N30] 32L25: Twistor theory, double fibrations [See also 53C28] 43A85: Analysis on homogeneous spaces 58J72: Correspondences and other transformation methods (e.g. Lie- Bäcklund) [See also 35A22]

**Keywords**

Penrose transform semisimple Lie group flag variety unitary representation Dolbeault cohomology hypergeometric function integral geometry symmetric space

#### Citation

SEKIGUCHI, Hideko. The Penrose transform for $Sp(n,R)$ and singular unitary representations. J. Math. Soc. Japan 54 (2002), no. 1, 215--253. doi:10.2969/jmsj/1191593961. https://projecteuclid.org/euclid.jmsj/1191593961