Text-to-image diffusion models are a class of deep generative models that have demonstrated impressive image generation capabilities. However, these models are susceptible to the implicit biases present in the webpage-scale text-image training pairs, which may not accurately model the aspects of images that we care about. This can lead to suboptimal samples, model biases, and images that are incongruent with human ethics and preferences. This work presents an effective and scalable algorithm that leverages reinforcement learning (RL) to improve diffusion models, encompassing a diverse range of reward functions such as human preference, coherence, and fairness, covering millions of images. We demonstrate how our method significantly outperforms existing approaches, aligning diffusion models with human preferences. We further illustrate how it substantially improves the pretrained Stable Diffusion (SD) model, resulting in samples preferred by humans by 80.3% while also enhancing the compositional and diversity of generated samples.