Abstract: We study the dephasing of two-electron states in a single quantum dot in both GaAs and Si. We investigate dephasing induced by electron-phonon coupling and by charge noise analytically for pure orbital excitations in GaAs and Si, as well as for pure valley excitations in Si. In GaAs, polar optical phonons give rise to the most important contribution, leading to GHz dephasing rates. For Si, intervalley optical phonons lead to typical dephasing rates of ~100 kHz for orbital excitations and ~1 MHz for valley excitations. For harmonic, disorder-free quantum dots, charge noise is highly suppressed for both orbital and valley excitations, since neither has an appreciable dipole moment to couple to electric field variations from charge fluctuators. However, both anharmonicity and disorder break the symmetry of the system, which can lead to increased dipole moments and therefore faster dephasing rates.