The opaque nature of AI/ML tools can significantly challenge data quality and decision-making. These tools can process data rapidly without human oversight, which may lead to errors that go unnoticed until they have larger consequences. For instance, a large language model once generated a seating chart that included a nonexistent name, a mistake that was only caught after invites were printed. This highlights the importance of ensuring data quality checks are in place to catch such issues early.

Organizations should prioritize the reproducibility of AI tool outputs. If different iterations of a test yield inconsistent results, it indicates potential issues with the AI product. Good AI tools should produce consistent and predictable results when tested on the same data. Evaluators must ensure that the results of AI evaluations are reproducible to maintain trust in the technology.

AI systems can extend and amplify biases, particularly if the training data reflects existing inequalities. For example, a tech company's AI for job candidate selection favored traits from a predominantly male workforce, leading to the automatic elimination of many female applicants. To mitigate these risks, organizations should carefully evaluate their training datasets for bias and implement guidelines to ensure fair and equitable AI outcomes.