The OCR 2.0 Model Is Here! Converting Charts, Geometric Shapes, and Musical Symbols into Editable Text

AIbase基地

Published inAI News · 5 min read · Oct 15, 2024

243

Title: OCR2.0: The Next-Generation Optical Character Recognition Model for Effortless Image-to-Text Conversion

Recently, researchers have developed a new universal Optical Character Recognition (OCR) model named GOT (General OCR Theory). In their paper, they introduced the concept of "OCR2.0" for the first time, aiming to combine the strengths of traditional OCR systems with the powerful capabilities of large language models.

The architecture of GOT is quite advanced, featuring an image encoder with approximately 80 million parameters and a decoder with 5 million parameters. The image encoder can compress images of 1024x1024 pixels into tokens, while the decoder is responsible for converting these tokens into text up to 8000 characters long. This way, the OCR2.0 model is capable of handling more than just simple text.

The charm of this new technology lies in its ability to recognize and convert various types of visual information, including English and Chinese scene text and document text, mathematical and chemical formulas, musical notations, simple geometric shapes, and charts with components. Such functionalities undoubtedly bring new possibilities for automation in fields such as science, music, and data analysis.

To optimize the training process, the research team first trained the encoder solely for text recognition tasks, then introduced Alibaba's Qwen-0.5B as the decoder and fine-tuned the model using diverse synthetic data. They generated millions of pairs of images and text for training data using rendering tools such as LaTeX, Mathpix-markdown-it, TikZ, Verovio, Matplotlib, and Pyecharts.

The modular design of GOT allows for flexible expansion of new features without retraining the entire model, significantly enhancing the system's update efficiency. Additionally, researchers report that GOT performs excellently across various OCR tasks, especially in document and scene text recognition, and even surpasses some specialized models and large language models in chart recognition.

It is worth noting that the research team has released the free demo and code of GOT on Hugging Face for others to use and further develop. This new model will undoubtedly drive the development of OCR technology and open up a broader range of applications.

Demo link: https://huggingface.co/spaces/stepfun-ai/GOT_official_online_demo

Key Points:
📌 GOT (General OCR Theory) is a new OCR model that combines traditional OCR systems with large language models, known as OCR2.0.
📌 The model can recognize and convert various visual information, including text, formulas, musical notations, and charts, applicable to a wide range of fields.
📌 The modular design and synthetic data training enable GOT to have flexible expansion capabilities and perform excellently in multiple OCR tasks.

OCR 2.0 Model: One-Click Conversion of Text, Formulas, and Chart Images to Editable Text

In the digital age, quickly converting text content from images into editable text is a common and important need. Now, the emergence of a new optical character recognition (OCR) model called GOT (General Optical Character Recognition Theory) marks the advancement of OCR technology into the 2.0 era. This innovative model combines the advantages of traditional OCR systems with large language models, aiming to create a more efficient and intelligent text recognition tool. The GOT model employs an innovative end-to-end architecture, a design that not only saves resources but also greatly enhances performance.

OpenAI Funding Negotiations: Apple Withdraws, Microsoft and Nvidia May Participate

OpenAI is currently in a round of funding negotiations expected to raise nearly $7 billion. According to insiders, Apple has recently withdrawn from participating in this round of talks, which are set to conclude next week. Another two tech giants, Microsoft and Nvidia, are also in discussions to join this round of negotiations. Insiders suggest that Microsoft is expected to invest an additional $1 billion on top of its previous investment of $13 billion in the company. The funding negotiations are not yet finalized, and participants and investment amounts may change. This round of financing will help OpenAI further advance.

Is Google in Trouble? Perplexity AI Plans Ultra-Low-Priced Advertising Model, Negotiating with Nike and Marriott

Recently, the much-watched AI startup Perplexity AI, backed by Amazon founder Jeff Bezos, has been negotiating with several well-known brands, including Nike and Marriott International, to launch a 'Sponsored Q&A' advertising system by the end of the year. This system aims to challenge Google's dominance in the $300 billion digital advertising market. Based in San Francisco, Perplexity AI is developing a brand-new advertising model that allows brands to bid on 'sponsored questions.'

Ordinary text recognition is outdated! GOT-OCR2.0 also understands formulas and sheet music

Recently, a model named GOT-OCR2.0 has attracted widespread attention in the industry. This model is capable of handling not only conventional text recognition tasks but also complex content such as formulas, tables, and sheet music, making it an all-around player in the OCR field. The core advantage of GOT-OCR2.0 lies in its diverse functionality and outstanding performance. Firstly, the model primarily supports the recognition of Chinese and English characters, and can be further fine-tuned to extend to more languages. This language adaptability enables GOT-OCR2.0 to be utilized in international applications.