SpaceByte is a brand new byte-level decoding architecture designed to address some drawbacks associated with the widely used Tokenization technique in large-scale language models. While Tokenization can significantly enhance model performance, it also introduces various defects such as performance bias, increased vulnerability to adversarial attacks, reduced character-level modeling effectiveness, and increased model complexity. Building upon the advantages of Tokenization, SpaceByte effectively resolves these issues. It leverages byte-level Transformers as the foundation and inserts larger Transformer blocks at model layers, particularly when encountering bytes that typically mark word boundaries such as spaces. Under the same training and inference computational resource budget, this architecture not only outperforms other byte-level models but also matches the performance of Tokenization-based Transformer models.