Recognizing familiar but innocuous stimuli and suppressing behavioral response to those stimuli are critical steps in dedicating cognitive resources to significant elements of the environment. Recent work in the visual system has uncovered key neocortical mechanisms of this familiarity that emerges over days. Specifically, exposure to phase-reversing gratings of a specific orientation causes long-lasting stimulus-selective response potentiation (SRP) in layer 4 of mouse primary visual cortex (V1) as the animal’s behavioral responses are reduced through habituation. This plasticity and concomitant learning require the NMDA receptor and the activity of parvalbumin-expressing (PV+) inhibitory neurons. Changes over the course of seconds and minutes have been less well studied in this paradigm, so we have here characterized cortical plasticity occurring over seconds and minutes, as well as days, to identify separable forms of plasticity accompanying familiarity. In addition, we show evidence of interactions between plasticity over these different timescales and reveal key mechanistic differences. Layer 4 visual-evoked potentials (VEPs) are potentiated over days, and they are depressed over minutes, even though both forms of plasticity coincide with significant reductions in behavioral response. Adaptation, classically described as a progressive reduction in synaptic or neural activity, also occurs over the course of seconds, but appears mechanistically separable over a second as compared to tens of seconds. Interestingly, these short-term forms of adaptation are modulated by long-term familiarity, such that they occur for novel but not highly familiar stimuli. Genetic knock-down of NMDA receptors within V1 prevents all forms of plasticity while, importantly, the modulation of short-term adaptation by long-term familiarity is gated by PV+ interneurons. Our findings demonstrate that different timescales of adaptation/habituation have divergent but overlapping mechanisms, providing new insight into how the brain is modified by experience to encode familiarity.