The Penrose transform for Sp(n,R) and singular unitary representations

Abstract

We give a general definition of the Radon-Penrose transform for a Zuckerman-Vogan derived functor module of a reductive Lie group $G$, which maps from the Dolbeault cohomology group over a pseudo-Kähler homogeneous manifold into the space of smooth sections of a vector bundle over a Riemannian symmetric space. Furthermore, we formulate a functorial property between two Penrose transforms in the context of the Kobayashi theory of discretely decomposable restrictions of unitary representations.

Based on this general theory, we study the Penrose transform for a family of singular unitary representations of $Sp(n,R)$ in details. We prove that the image of the Penrose transform is exactly the space of global holomorphic solutions of the system of partial differential equations of minor determinant type of odd degree over the bounded symmetric domain of type CI, which is biholomorphic to the Siegel upper half space. This system might be regarded as a generalization of the Gauss-Aomoto-Gelfand hypergeometric differential equations to higher order. We also find a new phenomenon that the kernel of the Penrose transform is non-zero, which we determine explicitly by means of representation theory.