Let $X$ be a complex smooth projective variety such that the exterior power of the tangent bundle $\bigwedge^{r} T_{X}$ is nef for some $1\leq r<\dim X$. We prove that, up to a finite étale cover, $X$ is a Fano fiber space over an Abelian variety. This gives a generalization of the structure theorem of varieties with nef tangent bundle by Demailly, Peternell and Schneider [5] and that of varieties with nef $\bigwedge^{2} T_{X}$ by the author [20]. Our result also gives an answer to a question raised by Li, Ou and Yang [15] for varieties with strictly nef $\bigwedge^{r} T_{X}$ when $r < \dim X$.