MathCoder is an open-source language model-based mathematical reasoning tool. By fine-tuning the model and generating high-quality datasets, it enables the alternation of natural language, code, and execution results, thereby enhancing mathematical reasoning capabilities. MathCoder has achieved the highest scores on both the MATH and GSM8K datasets, surpassing other open-source alternatives by a significant margin. MathCoder not only outperforms ChatGPT-3.5 and PaLM-2 on GSM8K and MATH but also exceeds GPT-4 on the competitive MATH dataset.