Harvard University announced on Thursday the release of a high-quality dataset containing nearly one million public domain books, which anyone can use to train large language models and other AI tools. This dataset was created by Harvard's newly established Institutional Data Initiative and is funded by Microsoft and OpenAI. The included books are all works scanned by the Google Books Project that are no longer under copyright protection.

The New York Times and the Daily News encountered an unexpected twist in their copyright lawsuit: an OpenAI engineer inadvertently deleted virtual machine search data that could have been key evidence, adding a dramatic turn to this high-profile legal dispute. According to a letter submitted to the U.S. District Court for the Southern District of New York on Wednesday night, lawyers and technical experts for the two media companies had previously invested over 150 hours searching OpenAI's AI training dataset. However, on November 14, an OpenAI engineer accidentally deleted data stored on the virtual machine.

LAION launched Re-LAION-5B, the world's first AI training dataset that fully removes links to CSAM, aimed at addressing the issue of Child Sexual Abuse Material (CSAM). This dataset has been significantly improved over LAION-5B and is mainly divided into two versions: Re-LAION-5B Research and Research-Safe. A total of 2,236 CSAM links have been removed, including 1,008 from child protection organizations' lists. The dataset contains 5.5 billion pairs of text and images, designed to help

Recently, the 'Diting' seismic wave large model, jointly developed by the National Supercomputing Center in Chengdu, the Institute of Geophysics of the China Earthquake Administration, and Tsinghua University, was officially released in Chengdu, Sichuan. This model is the first seismic wave large model in the country to reach 100 million parameters, marking a significant breakthrough in the integration of seismology research and artificial intelligence technology in China.

IBM opened the Granite13B LLM model for enterprise applications in May. Now, Armand Ruiz, the vice president of IBM's AI platform products, has publicly disclosed the full content of the comprehensive 6.48TB dataset used to train Granite13B.The dataset, after strict preprocessing, was reduced to 2.07TB, a reduction of 68%. Ruiz emphasizes that this step is crucial for ensuring a high-quality, unbiased, ethical, and legally compliant dataset to meet the needs of enterprise applications. The datas

ZhiYuan Research Institute has released a code generation training dataset called TACO, aimed at providing more challenging training data and evaluation benchmarks for code generation models. TACO has advantages in terms of data scale, quality, and evaluation schemes, including a larger training and testing set, diverse problem-solving answers, and fine-grained labels. Experimental results show that current popular code generation models show significant differences compared to GPT-4 in TACO evaluations, indicating that there is still room for improvement in this field. TACO is not just a challenging...

OpenAI has launched the Data Partnerships initiative to collaborate with third parties in creating public and private datasets for AI model training. OpenAI seeks data across multiple domains, languages, and formats to enhance the understanding of AI models. The initiative aims to collect various forms of data, including images, audio, and video, to reflect human society and achieve the goal of safer, more beneficial AI for humanity. OpenAI's initiative has sparked discussions about the rights of data owners and transparency.

The ZhiYuan Research Institute has released the world's largest Chinese-English semantic vector model training dataset, MTP, with a data scale of 300 million pairs. MTP is the largest open-source dataset of Chinese-English related text pairs, providing an important foundation for training semantic vector models. The dataset includes Chinese-English text pairs from multiple sources, covering various types such as Q&A, comments, and news. The ZhiYuan Research Institute stated that this data plays a crucial role in training large models and will promote collaborative innovation in artificial intelligence. The release of this dataset is expected to address the shortage of training datasets for Chinese models.

The anti-piracy organization successfully demanded the takedown of the AI training dataset 'Books3' from the online piracy book repository The Eye. The 'Books3' dataset consists of 37GB of text, which has been used to train artificial intelligence models by multiple companies. The anti-piracy organization stated that AI poses new challenges to copyright and plans to continue combating other websites hosting this dataset.