RAGElo is a toolkit that leverages the Elo rating system to help select the best-performing Large Language Model (LLM) agents enhanced with Retrieval Augmented Generation (RAG). While prototyping and integrating generative LLMs in production has become easier, evaluation remains the most challenging aspect of these solutions. RAGElo addresses this by comparing the answers of different RAG pipelines and prompts to multiple questions, calculating rankings for various setups. This provides a clear overview of which configurations are effective and which are not.