ImagenHub
ImagenHub: Inference and Evaluation of Standardized Conditional Image Generation Models
CommonProductImageConditional image generationModel evaluation
ImagenHub is a one-stop repository for standardizing the inference and evaluation of all conditional image generation models. The project first defines seven prominent tasks and creates high-quality evaluation datasets. Second, we build a unified inference pipeline to ensure fair comparisons. Third, we design two human evaluation metrics, semantic consistency and perceptual quality, and establish comprehensive guidelines for evaluating generated images. We train expert reviewers to evaluate model outputs based on the proposed metrics. This human evaluation achieved high inter-rater consistency on 76% of the models. We comprehensively evaluated around 30 models and observed three key findings: (1) The performance of existing models is generally unsatisfactory, with 74% of models scoring lower than 0.5 overall except for text-guided image generation and theme-driven image generation. (2) We examined claims made in published papers and found 83% of the claims to be accurate. (3) Apart from theme-driven image generation, existing automatic evaluation metrics have no Spearman correlation coefficient higher than 0.2. In the future, we will continue to evaluate newly released models and update the rankings to track the progress of the conditional image generation field.
ImagenHub Visit Over Time
Monthly Visits
8835
Bounce Rate
50.65%
Page per Visit
1.4
Visit Duration
00:00:16