PHF8 is a histone demethylase associated with X-linked mental retardation. It has been described as a transcriptional co-activator involved in cell cycle progression, but its physiological role is still poorly understood. Here we show that PHF8 controls the expression of genes involved in cell adhesion and cytoskeleton organization such as RhoA, Rac1 and GSK3β. A lack of PHF8 not only results in a cell cycle delay but also in a disorganized actin cytoskeleton and impaired cell adhesion. Our data demonstrate that PHF8 directly regulates the expression of these genes by demethylating H4K20me1 at promoters. Moreover, c-Myc transcription factor cooperates with PHF8 to regulate the analysed promoters. Further analysis in neurons shows that depletion of PHF8 results in down-regulation of cytoskeleton genes and leads to a deficient neurite outgrowth. Overall, our results suggest that the mental retardation phenotype associated with loss of function of PHF8 could be due to abnormal neuronal connections as a result of alterations in cytoskeleton function.