Centromeres are chromosomal loci marked by histone variant CenH3 (centromeric histone H3) and essential for genomic stability and cell division. The budding yeast E3 ubiquitin ligase Psh1 selectively recognizes the yeast CenH3 (Cse4) for ubiquitination and controls the cellular level of Cse4 for proteolysis, but the underlying mechanism remains largely unknown. Here, we show that Psh1 uses a Cse4-binding domain (CBD, residues 1-211) to interact with Cse4-H4 instead of H3-H4, yielding a dissociation constant (Kd) of 27 nM. Psh1 recognizes Cse4-specific residues in the L1 loop and α2 helix to ensure Cse4 binding and ubiquitination. We map the Psh1-binding region of Cse4-H4 and identify a wide range of Cse4-specific residues required for the Psh1-mediated Cse4 recognition and ubiquitination. Further analyses reveal that histone chaperone Scm3 can impair Cse4 ubiquitination by abrogating Psh1-Cse4 binding. Together, our study reveals a novel Cse4-binding mode distinct from those of known CenH3 chaperones and elucidates the mechanism by which Scm3 competes with Psh1 for Cse4 binding.