We demonstrate ultrafast plasmonic electron emission and acceleration from Ag nanolayers having different roughness. We obtained the spectrum of the electrons and found that the surface roughness deeply influences the properties of the electron spectra. Numerical simulations on propagating surface plasmons coupled to localized plasmons on surface grains support the observations. Applications related to ultrafast electron sources and ultrafast photocathodes are envisaged.