Nanodiamond integration with optical fibers has proved a compelling methodology for magneto-optics. We reveal that the applicability of nanodiamonds in nonlinear optics goes beyond the previous demonstrations of frequency converters. Instead, we exploit the recently reported volumetric integration of nanodiamonds along the optical fiber core and show that the nonlinear response of glasses can be manipulated by nanodiamonds. By taking the mature z-scan approach we measure the nonlinear absorption and nonlinear refraction of three dielectric materials containing nanodiamonds in different concentrations and sizes. The work begins with nanodiamond-water suspensions, which offer the advantage of rapidly assessing the dependence of the nonlinear refractive index on the nano-particle concentration and size. Subsequently we investigate two fiber preforms based on silica and soft glass doped with nanodiamonds to evaluate the feasibility of nonlinearity shaping. We achieve a nearly 20% reduction of the nonlinear refractive index of fused silica containing trace amounts of nanodiamonds relative to a pristine reference. The demonstration of such a noticeable impact on the nonlinear response of the key optical material widely accepted by ultrafast optics practitioners provides a guideline for future work on the novel concept of negative nonlinearity fibers, which could disrupt the established chromatic dispersion-nonlinearity landscape.