Follow-Your-Emoji is a diffusion model-based portrait animation framework that can animate target expression sequences onto reference portraits while maintaining the consistency of the portrait identity, the expressiveness of the emotions, the temporal coherence, and fidelity. By employing expression-aware landmark guidance and fine-grained facial loss techniques, the model significantly enhances its performance in controlling the freestyle portrait expressions, encompassing real people, cartoons, sculptures, even animals. Furthermore, its simple and effective progressive generation strategy extends to stable long-term animations, significantly increasing its potential applications.