In today's digital age, the issue of online safety for children is becoming increasingly important. Recently, Roblox, Discord, OpenAI, and Google launched a nonprofit organization called ROOST (Robust Open Online Safety Tools) aimed at building a scalable and interconnected safety infrastructure for the AI era. The objective of this new organization is to provide open-source safety tools to public and private entities to help them enhance protection on their platforms, with a particular focus on child safety. Source of the image: Image generated by AI.

Character AI, a platform that allows users to interact with AI chatbots for role-playing, recently filed a motion to dismiss a lawsuit brought by a teenager's parent in the U.S. District Court for the Middle District of Florida. The parent, Megan Garcia, alleges that Character AI's technology harmed her 14-year-old son, Sewell Setzer III, claiming that while interacting with a chatbot named 'Dany', he gradually became isolated from the real world and ultimately led to...

At the latest JP Morgan Healthcare Conference in 2025, NVIDIA and Innophore announced an exciting collaboration to launch an AI-driven drug safety screening platform named CavitOmiX by the end of January 2025. This platform aims to improve the efficiency and accuracy of drug design through the analysis of protein binding sites, marking a revolutionary transformation in the field of drug development. Christian Gruber, CEO of Innophore.

Recently, Nvidia announced the addition of three safety features on its NeMo Guardrails platform, aimed at helping businesses better manage and control AI chatbots. These microservices target common challenges in AI safety and content moderation, offering a range of practical solutions. Among them, the Content Safety service can review content before the AI responds to users, detecting any potential harmful information. This service helps prevent...

Recently, OpenAI showcased its more proactive red team testing strategy in the field of AI safety, surpassing its competitors, especially in the critical areas of multi-step reinforcement learning and external red team testing. The two papers released by the company establish new industry standards for enhancing the quality, reliability, and safety of AI models. The first paper, 'OpenAI's AI Model and System External Red Team Testing Methodology,' highlights the effectiveness of specialized external teams in identifying security vulnerabilities that internal testing may overlook. These external teams consist of cyber...

Mindgard has announced the completion of $8 million in funding aimed at accelerating its research and development efforts and expanding its presence in the U.S. market. The financing round was led by .406Ventures, with participation from Atlantic Bridge, Willowtree Investments, and existing investors IQ Capital and Lakestar. Additionally, Mindgard has appointed two industry leaders to take on the roles of Head of Product and Head of Marketing.

Recently, an incident in a Facebook mushroom enthusiasts group has once again raised concerns about the safety of AI applications. According to 404Media, an AI agent named FungiFriend infiltrated a northeastern mushroom identification and discussion group with 13,000 members and provided potentially fatal erroneous advice. When asked how to cook Sarcosphaera coronaria mushrooms, which contain high levels of arsenic, FungiFriend not only incorrectly stated that the mushroom...

French AI startup Mistral AI has officially launched a new content moderation API aimed at competing with OpenAI and other industry leaders, while addressing the growing concerns around AI safety and content filtering. This service is based on Mistral's Ministral8B model, finely tuned to detect potential harmful content across nine different categories, including pornography, hate speech, violence, dangerous activities, and personally identifiable information.

Recently, the results of an AI weapon scanning pilot project in New York City left many astonished. The scanning equipment provided by a tech company named Evolv failed to detect a single gun but flagged 12 knives instead. The image source note: Images generated by AI, Image licensed by service provider Midjourney. This trial, which started at the end of July, lasted only a month and produced disappointing results. According to CBS New York, although the equipment detected 12 knives, it did not result in any arrests.

In the current rapid development of artificial intelligence technology, Anthropic has recently announced an update to its 'Responsibility Expansion Policy (RSP)', which aims to effectively manage the potential risks posed by high-capability AI systems. As the company behind the popular chatbot Claude, this move by Anthropic clearly aims to find a balance between the continuously enhancing AI capabilities and the necessary safety standards. This new policy introduces what is called capability thresholds, as an additional safety assurance when AI model capabilities are increased.

Recently, California's artificial intelligence safety bill SB1047 has sparked intense competition between Hollywood and Silicon Valley. While it remains unclear whether California Governor Gavin Newsom will sign the bill, several high-profile celebrities have come forward to urge him to support the legislation. The image credit note states: Image generated by AI, image provided by Midjourney. This open letter has over 125 signatures from Hollywood stars, including Ava DuVernay, Jane Fonda, and J.J. Abrams. The letter states, 'We firmly believe that artificial intelligence...

Recently, U.S. officials announced that a global Artificial Intelligence (AI) Safety Summit will be hosted in San Francisco in November. As AI technology advances rapidly, governments around the world are striving to keep pace with this change. The summit will be co-chaired by U.S. Secretary of Commerce Gina Raimondo and Secretary of State Antony Blinken, and is scheduled for November 20-21. Raimondo stated in a release, 'As AI develops rapidly, we are doing everything we can.' She noted the importance of close international collaboration and thoughtful coordination.

Recently, OpenAI's CEO Sam Altman announced his decision to leave the company's internal safety committee, which was established in May of this year to oversee key safety and security decisions in the development and deployment of OpenAI's AI models. Altman's decision has drawn significant attention as he hopes to hand this role over to third-party experts to enhance transparency, collaboration, and security. With Altman's departure, this safety committee will transform into an 'independent oversight committee' focused on safety and security issues.

Recently, the US Department of Commerce released a proposal requiring artificial intelligence (AI) companies to provide detailed reports, aimed at stricter regulation of this rapidly evolving technological field. The notice, titled 'Proposed Rules,' was published on September 9, with the intention of ensuring the safety, reliability of AI technology, and preventing potential misuse by foreign adversaries or non-state actors. Photo credit: Image generated by AI, image licensed from Midjourney. Secretary of Commerce Gina Raimondo stated at a press conference that the rapid development of AI...