Nucleocytoplasmic transport occurs through the nuclear pore complex (NPC), which in yeast is a ~50 MDa complex consisting of ~30 different proteins. Small molecules can freely exchange through the NPC, but macromolecules larger than ~40 kDa must be aided across by transport factors, most of which belong to a related family of proteins termed karyopherins (Kaps). These transport factors bind to the disordered phenylalanine-glycine (FG) repeat domains in a family of NPC proteins termed FG nups, and this specific binding allows the transport factors to cross the NPC. However, we still know little in terms of the molecular and kinetic details regarding how this binding translates to selective passage of transport factors across the NPC. Here we show that the specific interactions between Kaps and FG nups are strongly modulated by the presence of a cellular milieu whose proteins appear to act as very weak competitors that nevertheless collectively can reduce Kap/FG nup affinities by several orders of magnitude. Without such modulation, the avidities between Kaps and FG nups measured in vitro are too tight to be compatible with the rapid transport kinetics observed in vivo. We modeled the multivalent interactions between the disordered repeat binding sites in the FG nups and multiple cognate binding sites on Kap, showing that they should indeed be sensitive to even weakly binding competitors; the introduction of such competition reduces the availability of these binding sites, dramatically lowering the avidity of their specific interactions and allowing rapid nuclear transport.