Refractory metal nitrides have recently gained attention in various fields of modern photonics due to their cheap and robust production technology, silicon-technology compatibility, high thermal and mechanical resistance, and competitive optical characteristics in comparison to typical plasmonic materials like gold and silver. In this work, we demonstrate that by varying the stoichiometry of sputtered nitride films, both static and ultrafast optical responses of refractory metal nitrides can efficiently be controlled. We further prove that the spectral changes in ultrafast transient response are directly related to the position of the epsilon-near-zero region. At the same time, the analysis of the temporal dynamics allows us to identify three time components: the “fast” femtosecond one, the “moderate” picosecond one, and the “slow” at the nanosecond time scale. We also find out that the non-stoichiometry does not significantly decrease the recovery time of the reflectance value. Our results show the strong electron-phonon coupling and reveal the importance of both the electron and lattice temperature-induced changes in the permittivity near the ENZ region and the thermal origin of the long tail in the transient optical response of refractory nitrides.