📖 tl;dr Responsible AI

The AI Risk Repository: A Comprehensive Meta-Review, Database, and Taxonomy of Risks From Artificial Intelligence

2 min read

Source: https://arxiv.org/abs/2408.12622

Summary

This paper presents the AI Risk Repository, a comprehensive effort to systematically analyze and categorize AI risks through a meta-review of existing taxonomies. The authors developed a living database of 777 risks extracted from 43 taxonomies, organized using two complementary frameworks: a high-level Causal Taxonomy and a mid-level Domain Taxonomy.

Key Points

Problem/Motivation

  • Lack of shared understanding of AI risks impedes comprehensive discussion, research, and response
  • Existing risk classifications are uncoordinated and inconsistent
  • Need for a common frame of reference for discussing and addressing AI risks

Methodology

  1. Systematic literature search yielding 43 relevant taxonomies
  2. Extraction of 777 distinct risks into a living database
  3. Development of two taxonomies through best-fit framework synthesis:
    • Causal Taxonomy: Classifies risks by entity (Human/AI), intentionality, and timing
    • Domain Taxonomy: Organizes risks into 7 domains and 23 subdomains

Key Findings

Causal Analysis

  • 51% of risks attributed to AI systems vs 34% to humans
  • Similar proportion of intentional (35%) vs unintentional (37%) risks
  • Most risks (65%) occur post-deployment vs pre-deployment (10%)

Domain Coverage

Most frequently covered domains:

  • AI system safety, failures & limitations (76% of documents)
  • Socioeconomic & environmental harms (73%)
  • Discrimination & toxicity (71%)

Underexplored areas:

  • AI welfare and rights (<1% of risks)
  • Competitive dynamics (1%)
  • Pollution of information ecosystem (1%)

Applications

  1. Policymakers

    • Aid in operationalizing vague concepts of “harm” and “risk”
    • Support development of compliance metrics
    • Facilitate international collaboration

  2. Auditors

    • Framework for comprehensive risk assessment
    • Support development of auditing standards

  3. Academics

    • Tool for research synthesis
    • Identification of research gaps
    • Support for education and training

  4. Industry

    • Assessment of potential risks in development plans
    • Tracking emerging risks
    • Foundation for risk management strategies

Limitations

  • Single reviewer for risk extraction and coding
  • Most source documents lack explicit risk definitions
  • Binary pre/post-deployment classification may oversimplify
  • Does not capture risk impact or likelihood
  • Focus primarily on language models rather than broader AI contexts

Future Work

  • Development of more granular categorizations
  • Addition of impact and likelihood dimensions
  • Exploration of underrepresented risk areas
  • Creation of more rigorous ontology
  • Extension to broader AI contexts beyond language models

Personal Notes

The AI Risk Repository represents a significant step toward creating a shared understanding of AI risks, though its utility will depend on continued maintenance and community adoption. The two-taxonomy approach (causal and domain) provides useful flexibility for different analytical needs.

Recent