Sketchpad enables language models to draw using lines, boxes, and markers, which is closer to human sketching and facilitates reasoning. Additionally, Sketchpad can utilize specialized visual models during the drawing process, such as using object detection models to create bounding boxes and segmentation models to draw masks, further enhancing visual perception and reasoning capabilities.

Shanghai AI Lab, together with Ruijin Hospital affiliated to Shanghai Jiao Tong University School of Medicine, has released the medical multimodal foundational model group 'PuYi 2.0'. PuYi 2.0 adds multi-domain models and incremental language parameters, covering various data modalities such as medical images, medical texts, and bioinformatics. New open-source datasets include the medical image segmentation dataset SA-Med2D-20M and the pathology dataset SNOW. PuYi 2.0 incorporates an evaluation module to provide reference implementations for medical model capabilities, delivering a one-stop open-source solution for large medical models.