Recently, an incident in a Facebook mushroom enthusiasts group has once again raised concerns about the safety of AI applications. According to 404Media, an AI agent named FungiFriend infiltrated a northeastern mushroom identification and discussion group with 13,000 members and provided potentially fatal erroneous advice. When asked how to cook Sarcosphaera coronaria mushrooms, which contain high levels of arsenic, FungiFriend not only incorrectly stated that the mushroom...

French AI startup Mistral AI has officially launched a new content moderation API aimed at competing with OpenAI and other industry leaders, while addressing the growing concerns around AI safety and content filtering. This service is based on Mistral's Ministral8B model, finely tuned to detect potential harmful content across nine different categories, including pornography, hate speech, violence, dangerous activities, and personally identifiable information.

Recently, the results of an AI weapon scanning pilot project in New York City left many astonished. The scanning equipment provided by a tech company named Evolv failed to detect a single gun but flagged 12 knives instead. The image source note: Images generated by AI, Image licensed by service provider Midjourney. This trial, which started at the end of July, lasted only a month and produced disappointing results. According to CBS New York, although the equipment detected 12 knives, it did not result in any arrests.

In the current rapid development of artificial intelligence technology, Anthropic has recently announced an update to its 'Responsibility Expansion Policy (RSP)', which aims to effectively manage the potential risks posed by high-capability AI systems. As the company behind the popular chatbot Claude, this move by Anthropic clearly aims to find a balance between the continuously enhancing AI capabilities and the necessary safety standards. This new policy introduces what is called capability thresholds, as an additional safety assurance when AI model capabilities are increased.

Recently, California's artificial intelligence safety bill SB1047 has sparked intense competition between Hollywood and Silicon Valley. While it remains unclear whether California Governor Gavin Newsom will sign the bill, several high-profile celebrities have come forward to urge him to support the legislation. The image credit note states: Image generated by AI, image provided by Midjourney. This open letter has over 125 signatures from Hollywood stars, including Ava DuVernay, Jane Fonda, and J.J. Abrams. The letter states, 'We firmly believe that artificial intelligence...

Recently, U.S. officials announced that a global Artificial Intelligence (AI) Safety Summit will be hosted in San Francisco in November. As AI technology advances rapidly, governments around the world are striving to keep pace with this change. The summit will be co-chaired by U.S. Secretary of Commerce Gina Raimondo and Secretary of State Antony Blinken, and is scheduled for November 20-21. Raimondo stated in a release, 'As AI develops rapidly, we are doing everything we can.' She noted the importance of close international collaboration and thoughtful coordination.

Recently, OpenAI's CEO Sam Altman announced his decision to leave the company's internal safety committee, which was established in May of this year to oversee key safety and security decisions in the development and deployment of OpenAI's AI models. Altman's decision has drawn significant attention as he hopes to hand this role over to third-party experts to enhance transparency, collaboration, and security. With Altman's departure, this safety committee will transform into an 'independent oversight committee' focused on safety and security issues.

Recently, the US Department of Commerce released a proposal requiring artificial intelligence (AI) companies to provide detailed reports, aimed at stricter regulation of this rapidly evolving technological field. The notice, titled 'Proposed Rules,' was published on September 9, with the intention of ensuring the safety, reliability of AI technology, and preventing potential misuse by foreign adversaries or non-state actors. Photo credit: Image generated by AI, image licensed from Midjourney. Secretary of Commerce Gina Raimondo stated at a press conference that the rapid development of AI...

The United States, the United Kingdom, and the European Union have jointly signed the first legally binding AI treaty, the 'AI Framework Convention', aimed at ensuring that AI applications adhere to principles of human rights, democracy, and the rule of law. The treaty requires signatory countries to follow principles that protect user data, respect the law, and operate transparently, implementing these principles through legislation and administrative measures. Although compliance with the treaty primarily relies on monitoring, and enforcement may be weak, it is seen as an important blueprint for countries to formulate AI laws. Currently, the US, EU, UK, and several other countries have signed, and the treaty will come into force in five...

Safe Superintelligence (SSI) was founded by former OpenAI Chief Scientist Ilya Sutskever, aiming to develop safe AI systems that surpass human capabilities. The company has raised $1 billion and is valued at $5 billion, focusing on the research and development of AI safety and superintelligent technologies rather than rushing to market. SSI values integrity and potential in its hiring principles, aiming to ensure that AI technology serves humanity and reduces risks through innovative thinking.

Highwire has launched an AI-powered safety risk management tool that provides contractors with deep insights through automated document reviews and risk identification, reducing thousands of man-hours spent on traditional manual reviews. The tool offers more accurate contractor assessments, helping managers implement mitigation strategies before work begins, categorizing safety protocols as general, detailed, or comprehensive, and going beyond simple pass/fail analysis to support more informed decision-making. This AI technology was developed at Harvard University and aims to serve large companies with mature safety programs as well as small contractors, with the goal of enhancing safety.

The 'Self-Regulatory Initiative for the Generative Artificial Intelligence Industry' was launched at the 2024 China Internet Civilization Conference in Chengdu, initiated by the Artificial Intelligence Safety Governance Committee of the China Cybersecurity Association in collaboration with industry, academia, and research institutions. The initiative focuses on five core areas: compliance and safety of data and algorithm models to ensure legal and secure data usage; content ecosystem development to encourage positive content generation and maintain a healthy online environment; technological innovation and quality enhancement to advance technology and quality assessment; ethical standards that respect human rights and social values; and communication, cooperation, and openness.

Changes at OpenAI have sparked broad concern in the industry regarding AI safety issues. The AGI safety team has significantly reduced in size, from about 30 members to around 16, raising worries about whether the company is neglecting AI safety concerns. The departures were not an organized effort, but rather a gradual loss of confidence among team members. Former governance researcher at OpenAI, Daniel Kokotajlo, noted that the reduction in the team appears to be related to the company's increased focus on product and commercialization development. In response to external doubts, OpenAI emphasizes its commitment to safety.

California lawmakers have passed the SB1047 bill aimed at ensuring AI safety, which is set to be signed by Governor Gavin Newsom. The bill requires AI models with training costs exceeding $100 million to undergo safety testing before release and establishes an emergency stop function in case of accidents. It also mandates reporting to the California Attorney General within 72 hours of an incident and establishes a 'Cutting-edge Model Bureau' to oversee compliance, with substantial fines for violators. This initiative aims to prevent AI misuse, ensure safety, and has garnered support from notable figures in the AI field. However, tech giants have had mixed reactions.

Google has announced the reactivation of the character image generation feature for its AI chatbot Gemini, aimed at producing diverse content including photo-realistic landscapes and textured paintings. This feature is powered by Google's AI text-to-image generator Imagen3 and currently supports only English. It will be available to Gemini Advanced, Business, and Enterprise users in the coming days. The newly upgraded Imagen3 model offers improved performance and includes built-in safety measures optimized to avoid historical inaccuracies.