California Attorney General Issues Warning on Artificial Intelligence in Health Care

On January 13, 2025, California Attorney General Rob Bonta (the California AG) issued two Legal Advisories regarding the utilization of artificial intelligence (AI). The first Legal Advisory provides guidance to consumers and entities developing, selling, and using AI describing their rights and obligations under California law. The second Legal Advisory, entitled Application of Existing California Law to Artificial Intelligence in Healthcare (the Health Care Legal Advisory), is expressly directed at health care providers, insurers, vendors, investors, and other health care entities who develop, sell, and use AI and other automated decision-making tools. In the accompanying press release, the California AG notes that: 

AI might be changing, innovating, and evolving quickly, but the fifth largest economy in the world is not the wild west; existing California laws apply to both the development and use of AI. Companies, including healthcare entities, are responsible for complying with new and existing California laws and must take full accountability for their actions, decisions, and products.

The California AG’s Concerns with AI in Health Care

The California AG has previously expressed concerns regarding the increased utilization of AI and algorithmic tools and the potential violation of California law. In August 2022, the California AG sent letters to 30 California hospitals requesting information regarding how the hospitals identify and address racial and ethnic disparities that may exist in their commercial decision-making tools. (See California AG press release here). The California AG raised the issue that such algorithmic tools, which often leverage AI, might have discriminatory impacts based on race and ethnicity.

The recent Legal Advisories highlight the California AG’s ongoing concerns. On the one hand, the Health Care Legal Advisory notes the potential benefits of AI-enabled tools including improving both patient and population health, reducing administrative burdens, increasing health equity, and facilitating appropriate information sharing. On the other hand, the risks of AI in health care include potential denials of necessary care, misallocation of health care resources, discrimination, and interference with patient autonomy and privacy.

Compliance with Existing Laws

The Health Care Legal Advisory discusses compliance with existing California consumer protection, unfair competition, civil rights, and patient privacy laws. First, the Health Care Legal Advisory outlines the broad scope of California’s Unfair Competition Law (Cal. Bus. & Prof. Code § 17200) which protects consumers against false advertising, marketing, and anticompetitive practices among other legal protections. 

Second, the Health Care Legal Advisory highlights California’s anti-discrimination law applicable broadly to individuals and entities receiving “state support” (Cal. Gov. Code § 11135). In this context, the California AG warns that while AI and other automated decision-making tools may be facially neutral, health care entities may not ignore or avoid data indicating racial, gender, or other protected classification inequities.

Third, the California AG notes that in some cases California medical privacy laws are more protective than federal law. As such, California health care entities may have a heightened obligation to preserve the confidentiality of medical information, ensure patient access to their medical information, and obtain robust patient informed consents. Further, the Health Care Legal Advisory posits that California health care providers consider whether disclosure of the utilization of AI and other automated decision-making tools should be included in such informed consents.

Lastly and notably, the Health Care Legal Advisory references California’s long-standing prohibition against the practice of medicine by corporations and other “artificial legal entities” (Cal. Bus. & Prof. Code § 2400). AI and other automated decision-making tools might violate this prohibition in instances where the tools interfere with professional medical judgment regarding patient care and treatment.

Compliance with Recently Enacted Laws

The Health Care Legal Advisory also mentions compliance with newly enacted California laws specifically directed towards AI. In particular, the Legal Advisory highlights SB-1120 prohibiting health care services plans from utilizing AI and other automated decision-making tools in making final medical necessity determinations to delay, deny, or modify care. Under SB-1120, licensed health care professionals competent to evaluate the specific clinical issues involved must make these medical necessity assessments. (See our Health Care Legislative Update here). 

Potentially Unlawful Activities

Under these existing and newly enacted legal authorities, the California AG provides examples of activities that may be unlawful in California when AI or other automated decision-making tools are utilized including:

• Denying health insurance claims in a manner that overrides a physician’s determination regarding necessary treatment.
• Determining patient access to health care based upon predictions utilizing a patient’s past health claims data and having a discriminatory disparate impact.
• Drafting patient notes, communications, or medical orders that include erroneous or misleading information.

Takeaways

The Health Care Legal Advisory confirms that the California AG will continue monitoring the increasing utilization of AI in the health care sector. As such, health care providers, insurers, vendors, and investors should continue to test, validate, and audit AI and other automated tools deployed in health care decision-making. Health care industry participants also should understand how the AI tools they utilize are trained, the quality and type of information that is considered, and how outputs and recommendations are generated. Lastly, health care entities should consider being transparent with patients regarding how AI is being utilized in making health care related decisions.

Subscribe To Viewpoints