Neuroendocrine prostate cancer (NEPC) is an aggressive variant of prostate cancer that either develops de novo or arises from prostate adenocarcinoma as a result of treatment resistance. Although the prostate basal cells have been shown to directly generate tumor cells with neuroendocrine features when transduced with oncogenic signaling, the identity of the cell-of-origin for de novo NEPC remains unclear. We show that the TACSTD2high human prostate luminal epithelia cells highly express SOX2 and are relatively enriched in the transition zone prostate. Both TACSTD2high and TACSTD2low luminal cells transduced by constitutively activated AKT1 (caAKT1), and c-Myc can form organoids containing versatile clinically relevant tumor cell lineages with regard to the expression of AR and the neuroendocrine cell markers Synaptophysin and Chromogranin A. Tumor organoid cells derived from the TACSTD2high luminal cells are more predisposed to neuroendocrine differentiation along passaging and are relatively more castration-resistant. Knocking down TACSTD2 and SOX2 both attenuate neuroendocrine differentiation of tumor organoid cells. This study demonstrates de novo neuroendocrine differentiation of the human prostate luminal epithelial cells induced by caAKT1 and c-Myc and reveals an impact of cellular status on initiation of lineage plasticity.