Social sciences are engaging in efforts to scrutinize the very processes of creation of scientific knowledge. Whereas much of the attention is revolving around questionable or suboptimal practices of data analysis, how data is collected from human participants received less scrutiny. Data collection in psychological, sociological, and management sciences is dramatically changing, moving progressively away from physical environments and towards the Internet. Researchers can now crowdsource data collection, recruiting and compensating people on Internet marketplaces (e.g., Mechanical Turk) for participating in their investigations. Today, it is customary to read articles in the highest ranked journals that are partially or entirely based on crowdsourced data. Crowdsourced investigations are inherently characterized by a lack of controllability over the research process. Researchers have little control over the sampling and execution stages of research, and largely ignore how their recruitment and design decisions affect the characteristics of the samples they collect and of the data they obtain from such samples. This poses unique challenges to the scientific validity of crowdsourced investigations that are currently unaccounted for. This project investigates these challenges, exploring how the characteristics and the quality of crowdsourced data depend on features of the crowdsourcing process that are universally relevant across research paradigms and crowdsourcing platforms. In particular, I will focus on the role of the incentives used to compensate participants, of the time (in the day, in the week) in which research is crowdsourced, and of participants research history. This will result in theoretically grounded and practically actionable insights to conduct more valid crowdsourced investigations. The Internet democratized science by lowering the barriers to its consumption and dissemination. Crowdsourcing can now allow a more democratic production of scientific knowledge. However, crowdsourcing has many undetected pitfalls. My ultimate goal is to contribute to better science by improving the quality of online research.