Noble metal nanolayers on flat substrates are often deposited with the use of semiconductor interlayers, which may strongly interact with the noble metal overlayer. We investigated the crystallinity, atomic concentration profile and optical parameters of ≈35 nm-thick silver and gold layers deposited on glass substrates with 2 nm-thick tellurium or selenium interlayers. Our study, based on X-ray reflectometry (XRR), X-ray diffraction (XRD), X-ray photoelectron spectroscopy (XPS) and ellipsometric measurements, showed that using either of these interlayers introduces strain in nanocrystals of both plasmonic films. This, in turn, influences the migration of Se and Te into the metal layers. Selenium atoms migrate both in the silver and gold nanolayers, while tellurium atoms migrate only in silver. The Te concentration curve clearly suggests that this migration is an effect of the segregation of Te atoms in the silver structure. The differences in crystallinity, as well as the migration process, strongly influence the optical parameters of Ag and Au. In the permittivity of Ag deposited on either Te or Se, additional plasmonic bands originating from grain boundary segregation or diffusion occur, while for the Au layer, such resonances were not pronounced. In the permittivity of both materials, the intensity of the interband transition peaks is strongly altered, possibly due to the nano-alloy formation, but more likely due to the microstrain on metal grains.