We consider the averaging principle for deterministic and stochastic systems with a fast stochastic component (family of continuous time Markov chains depending on the state of the system as a parameter). We show that, due to the nontrivial structure of the simplex of invariant probability measures of the chains, the limiting system should be considered on a graph with certain gluing conditions at the vertices of the graph.