Let $G$ be a group. An automorphism $\alpha$ of $G$ is called a commuting automorphism if $[\alpha(g),g]=1$ for all $g \in G$. Let $A(G)$ denote the set of all commuting automorphisms of $G$. A group $G$ is said to be an $A(G)$-group if $A(G)$ forms a subgroup of ${\rm Aut}(G)$, where ${\rm Aut}(G)$ denotes the group of all automorphisms of $G$. In [Proc. Japan Acad. Ser. A Math. Sci. 91 (2015), no. 5, 57-60] Rai proved that a finite $p$-group $G$ of co-class 2 for an odd prime $p$ is an $A(G)$-group. We prove that a finite $p$-group $G$ of co-class 3 for an odd prime $p$, under some conditions, is an $A(G)$-group.