A substantial proper submanifold $M$ of a Riemannian symmetric space $S$ is called a curved Lie triple if its tangent space at every point is invariant under the curvature tensor of $S$, i.e. a sub-Lie triple. E.g. any complex submanifold of complex projective space has this property. However, if the tangent Lie triple is irreducible and of higher rank, we show a certain rigidity using the holonomy theorem of Berger and Simons: $M$ must be intrinsically locally symmetric. In fact we conjecture that $M$ is an extrinsically symmetric isotropy orbit. We are able to prove this conjecture provided that a tangent space of $M$ is also a tangent space of such an orbit.