SH3 domains are peptide recognition modules that mediate the assembly of diverse biological complexes. We scanned billions of phage-displayed peptides to map the binding specificities of the SH3 domain family in the budding yeast, Saccharomyces cerevisiae. Although most of the SH3 domains fall into the canonical classes I and II, each domain utilizes distinct features of its cognate ligands to achieve binding selectivity. Furthermore, we uncovered several SH3 domains with specificity profiles that clearly deviate from the two canonical classes. In conjunction with phage display, we used yeast two-hybrid and peptide array screening to independently identify SH3 domain binding partners. The results from the three complementary techniques were integrated using a Bayesian algorithm to generate a high-confidence yeast SH3 domain interaction map. The interaction map was enriched for proteins involved in endocytosis, revealing a set of SH3-mediated interactions that underlie formation of protein complexes essential to this biological pathway. We used the SH3 domain interaction network to predict the dynamic localization of several previously uncharacterized endocytic proteins, and our analysis suggests a novel role for the SH3 domains of Lsb3p and Lsb4p as hubs that recruit and assemble several endocytic complexes.