Abstract Background Melanoma cancer causes serious health problem worldwide because of its rapid invasion to other organs and lack of satisfactory chemotherapy. The pGI50 anticancer activity values of 70 compounds from the NCI (National Cancer Institute) on MALME-3M cell line was modeled to describe the quantitative structure-activity relationships (QSARs) of the compounds, and some selected compounds were docked. Results The generated QSAR model was found to be statistically significant based on the obtained values of the validation keys such as R2 (0.885), $$ {R}_{\mathrm{adjusted}}^2 $$ R adjusted 2 (0.868), Q2cv (0.842), and $$ {R}_{pred}^2 $$ R pred 2 (0.738) required to evaluate the strength and robustness of QSAR model. Compound 39 was selected as a template due to its good pGI50 (9.205) and was modified to design new potent compounds. The predicted pGI50 activity of the designed compounds by the built model was N1 (9.836), N2 (12.876), N3 (10.901), and N4 (11.263) respectively. These proposed compounds were docked with V600E-BRAF receptor and the result shows that, N1, N2, N3, and N4 with free binding energy (FBE) of − 11.7 kcal mol−1, − 12.8 kcal mol−1, − 12.7 kcal mol−1, and − 12.9 kcal mol−1 respectively were better than the parent structure of the template (compound 39, FBE = − 7.0 kcal mol−1) and the standard V600E-BRAF inhibitor (Vemurafenib, FBE = − 11.3 kcal mol−1). Additionally, these compounds passed the drug-likeness criteria successfully to be orally bioavailable. Conclusion The proposed compounds were considered optimal as their performances are comparable to vemurafenib and possessed enhanced physicochemical properties. Thus recommends further research such as synthesis, in vivo, and ex-vivo evaluation.