We experimentally investigate the spectro-temporal characteristics of coherent supercontinuum (SC) pulses generated in several implementations of silica and soft-glass all-normal dispersion (ANDi) photonic crystal fibers optimized for pumping with Erbium (Er):fiber femtosecond laser technology. We characterize the resulting SC using time-domain ptychography, which is especially suitable for the measurement of complex, spectrally broadband ultrashort pulses. The measurements of the ANDi SC pulses reveal intricate pulse shapes, considerable temporal fine structure, and oscillations on time scales of < 25 femtoseconds, which differ from the smoothness and simplicity of temporal profiles obtained in numerical simulations and observed in previous experiments. We link the measured complex features to temporal sub-structures of the pump pulse, such as pre- and post-pulses and low-level pedestals, which are common in high pulse energy ultrafast Er:fiber systems. We also observe spectro-temporal structures consistent with incoherent noise amplification in weakly birefringent fiber samples. Our results highlight the importance of the pump source and polarization-maintaining (PM) fibers for high-quality SC generation and have practical relevance for many ultrafast photonics applications employing ANDi fiber-based SC sources.