The collective activity pattern of retinal ganglion cells, the retinal code, underlies higher visual processing. How does the ambient illuminance of the visual scene influence this retinal output? We recorded from isolated mouse and pig retina and from mouse dorsal lateral geniculate nucleus in vivo at up to seven ambient light levels covering the scotopic to photopic regimes. Across each luminance transition, most ganglion cells exhibited qualitative response changes, whereas they maintained stable responses within each luminance. We commonly observed the appearance and disappearance of ON responses in OFF cells and vice versa. Such qualitative response changes occurred for a variety of stimuli, including full-field and localized contrast steps and naturalistic movies. Our results suggest that the retinal code is not fixed but varies with every change of ambient luminance. This finding raises questions about signal processing within the retina and has implications for visual processing in higher brain areas.