Let X be a proper, smooth, geometrically connected curve of genus g≥2 over a p-adically complete discrete valuation field K. By the Albanese morphism withrespect to a given K-rational point, the curve X can be embedded into its Jacobian variety J. Then, assuming that J has ordinary semistable reduction, we prove that the inertia group of K acts trivially on the set of torsion points of J which lie on X, under certain mild conditions. As an application, we prove that the modular curve X₀(N) (N: a prime number greater than or equal to 23), embedded into its Jacobian variety by using a cusp, contains no torsion points other than the cusps (resp., the cusps and the Weierstrass points), if N∉{23,29,31,41,47,59,71} (resp., N∈{23,29,31,41,47,59,71}). This affirmatively answers a question posed by R. Coleman, B. Kaskel, and K. Ribet [CKR].