Score Distillation Sampling (SDS) is a recently popular method that relies on image diffusion models to control optimization problems with text prompts. This paper conducts an in-depth analysis of the SDS loss function, identifies inherent problems in its formulation, and proposes an unexpected yet effective fix. Specifically, we decompose the loss into different factors and isolate the component that generates noisy gradients. In the original formulation, high text guidance is used to account for noise, leading to undesirable side effects. Instead, we train a shallow network to mimic the time-step-dependent denoising insufficiency of the image diffusion model, effectively decoupling it. We demonstrate the versatility and effectiveness of our novel loss formulation through multiple qualitative and quantitative experiments, including optimized image synthesis and editing, zero-shot image translation network training, and text-to-3D synthesis.