The CB1 receptor mediates the central nervous system response to cannabinoids, and is a drug target for pain, anxiety and seizures. CB1 also responds to allosteric modulators, which influence cannabinoid binding and efficacy. To understand the mechanism of these compounds, we solved the crystal structure of CB1 with the negative allosteric modulator (NAM) ORG27569 and the agonist CP55940. The structure reveals that the NAM binds to an extrahelical site within the inner leaflet of the membrane, which overlaps with a conserved site of cholesterol interaction in many G protein-coupled receptors (GPCRs). The ternary structure with ORG27569 and CP55940 captures an intermediate state of the receptor, in which aromatic residues at the base of the agonist-binding pocket adopt an inactive conformation despite the large contraction of the orthosteric pocket. The structure illustrates a potential strategy for drug modulation of CB1 and other class A GPCRs.