Glyph-ByT5 is a custom text encoder aimed at improving the accuracy of visual text rendering in text-to-image generation models. It achieves this by fine-tuning a character-aware ByT5 encoder and utilizing a carefully curated dataset of paired glyph text. Integrating Glyph-ByT5 with SDXL results in the Glyph-SDXL model, enhancing text rendering accuracy in image design generation from below 20% to nearly 90%. This model also enables automatic multi-line layout rendering for paragraph text, maintaining high spelling accuracy for character counts ranging from dozens to hundreds. Furthermore, by fine-tuning on a small set of high-quality real images containing visual text, Glyph-SDXL has significantly improved its scene text rendering capability in open-domain real images. These encouraging results aim to encourage further exploration of designing custom text encoders for various challenging tasks.