The trans-Golgi network (TGN) is the main secretory pathway sorting station, where cargoes are packed into appropriate transport vesicles targeted to specific destinations. Exomer is a cargo adaptor necessary for direct transport of a subset of cargoes from the TGN to the plasma membrane in yeast. Here, we show that unlike classical adaptor complexes, exomer is not recruited en bloc to the TGN, but rather assembles through a stepwise pathway, in which first the scaffold protein Chs5 and then the cargo-binding units, the ChAPs, are recruited. Although all ChAPs are able to assemble functional exomer complexes, they do so with different efficiencies. The mutual relationship between ChAPs varies from cooperation to competition depending on their expression levels and affinities to Chs5 allowing an optimized and efficient cargo transport. The multifactorial assembly pathway results in an exquisitely fine-tuned adaptor complex, enabling the cell to quickly respond and adapt to changes such as stress.