On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, a Dallas-based artificial intelligence technology company that markets its software to hospitals and other health care facilities.
This settlement is important because:
- This is the first state attorney general settlement involving deceptive marketing of AI software.
- Although the settlement does not include a monetary payment, the Texas AG imposed heightened requirements on Pieces’ future marketing of its AI software.
- The settlement is an important reminder that states do not need new legislation to regulate the use of AI.
AI Software Company Allegedly Overstated Software’s Accuracy
According to the settlement, Pieces designed its generative AI product to assist medical staff with patient care, including summarizing, charting and drafting clinical notes. Pieces provided its software to at least four Texas hospitals, after Pieces allegedly made deceptive statements about the accuracy and safety of the products.
In particular, the Texas Attorney General alleged that Pieces advertised metrics to indicate that its products are “highly accurate” and had an error rate or “severe hallucination rate” of “<1 per 100,000.” The Texas AG, however, alleged that these metrics were most likely inaccurate and deceptive in violation of the Texas consumer protection law.
Settlement Requires Limitations and Disclosures for Ongoing Marketing
Pieces agreed to an assurance of voluntary compliance (AVC) imposing heightened requirements on its future marketing, including:
- All Pieces advertising will clearly and conspicuously disclose the definition of the metrics it uses to support its accuracy claims and the methods by which it calculates the metrics.
- Pieces is prohibited from making false, misleading or unsubstantiated representations about the accuracy, functionality, purpose or other features of its products.
- Pieces will clearly and conspicuously disclose any known or reasonably knowable harmful or potentially harmful uses or misuses of its products in writing to all current and future customers.
These disclosures are meant to ensure that hospital staff understand the extent to which they can rely on Pieces generative AI products. Under Texas law, a violation of the commitments in the AVC constitutes prima facie evidence of a violation of the Texas Consumer Protection Act.
Settlement Has Important Implications for AI Companies
The settlement is an important reminder that states do not need new legislation to regulate the use of AI, because consumer protection, privacy and antidiscrimination laws already apply to AI systems. In June, the Texas AG had announced a new data privacy and security initiative focusing on data privacy and AI enforcement under several new and old statutes, which has already resulted in an enforcement sweep involving more than 100 companies. The Massachusetts Attorney General’s Office issued guidance in April warning that misrepresenting the reliability, manner of performance, safety or condition of an AI system, including unsubstantiated claims about its capabilities, constitutes a violation of the Massachusetts consumer protection law.
Manatt will continue to follow these developments closely as state regulators provide additional guidance and other states, including California, advance AI-related legislation with a significant impact on health care.