While free electrons in metals respond to ultrafast excitation with refractive index changes on femtosecond time scales, typical relaxation mechanisms occur over several picoseconds, governed by electron-phonon energy exchange rates. Here, we propose tailoring these intrinsic rates by engineering a non-uniform electron temperature distribution through nanostructuring, thus, introducing an additional electron temperature relaxation channel. We experimentally demonstrate a sub-300 fs switching time due to the wavelength dependence of the induced hot electron distribution in the nanostructure. The speed of switching is determined by the rate of redistribution of the inhomogeneous electron temperature and not just the rate of heat exchange between electrons and phonons. This effect depends on both the spatial overlap between control and signal fields in the metamaterial and hot-electron diffusion effects. Thus, switching rates can be controlled in nanostructured systems by designing geometrical parameters and selecting wavelengths, which determine the control and signal mode distributions.