ChatQA is a series of conversational question answering (QA) models that can achieve GPT-4 level accuracy. We propose a two-stage instruction tuning method that can significantly improve the zero-shot conversational QA results of large language models (LLMs). To handle retrieval in dialogue-based QA, we fine-tuned dense retrievers on multi-turn QA datasets, which provides comparable results to using state-of-the-art query rewrite models while significantly reducing deployment costs. Notably, our ChatQA-70B outperforms GPT-4 (54.14 vs. 53.90) on the average score across 10 conversational QA datasets without relying on any synthetic data from the OpenAI GPT model.