Artificial intelligence (AI) is becoming increasingly influential in STEM education. AI tools can now support automated scoring, personalized feedback, tutoring and learning analytics. However, many advanced AI systems, including deep learning models and large language models (LLMs), operate like “black boxes”. They can provide scores or feedback, but it is difficult for educators, students and researchers to understand how AI-generated decisions are made.

In a recently published open-access book, Artificial Intelligence for STEM Education Research, a chapter “Explainable AI in STEM Education Research” is devoted to this issue. It examines how AI systems can be made more transparent, interpretable and useful for STEM education by introducing key explainable AI approaches and discussing how they can support trust, diagnostic insight, instructional decision-making, and responsible AI integration in STEM learning contexts.

Explainable AI, often called XAI, refers to methods and principles that help people understand, trust and potentially control the reasoning behind an AI model’s predictions. In education, explainability is not only a technical concern, but also a pedagogical and ethical concern.

When AI is used for scoring or feedback generation, educators need to know whether the system is focusing on meaningful evidence of learning. Students also benefit when AI feedback helps them understand why a response is strong, incomplete, or inaccurate. Researchers need explainability to evaluate whether AI models align with learning theories, assessment frameworks and disciplinary expectations.

Some AI models are easier to understand by design, such as linear and tree-based models. A linear model can help show how one factor, such as study time or prior performance, is associated with a predicted outcome. A tree-based model can show a sequence of decision rules that lead to a result. These models make the logic of AI prediction visible.

Other methods are designed to explain more complex AI systems. One approach uses “what-if” reasoning, also known as counterfactual explanation, to turn a prediction into guidance. For example, an AI tool might show how a student’s response could improve if they added a key scientific idea or corrected a misconception. This type of explanation can turn an AI score into useful feedback.

Another approach helps identify which parts of a student’s work mattered most to the AI. For example, in automated scoring, explainable AI can help show whether the system recognized important ideas such as photosynthesis or energy conversion in student responses. This helps teachers determine whether the AI is paying attention and detecting meaningful disciplinary content.

ChatGPT and other LLMs raise another important issue. Because these tools respond to prompts written in everyday language, the way users ask questions matter. Clear prompts, relevant examples, background information, and step-by-step reasoning can make AI interactions more visible and easier to refine. Strategies such as in-context examples, knowledge augmentations, and chain-of-thought prompting can help guide LLMs toward responses that are more interpretable and useful for teaching and learning.

Overall, a central message is that AI in STEM education should not be judged by accuracy alone. A score, recommendation, or feedback message becomes more valuable when people understand the reasoning behind it. XAI can help connect model outputs to the kinds of evidence educators care about, such as student reasoning, misconceptions, disciplinary understanding and progress over time.

As AI continues to evolve, its role in STEM education will require ongoing collaboration among AI researchers, educators, learning scientists and assessment experts to ensure that AI tools remain human-centered. Though viewed as a transformation within the education system, its success will depend on how thoughtfully it is implemented, with extreme importance that educators remain at the center of decisions about how AI is used, what evidence matters and how AI-generated insights should inform teaching and learning. Overall, if used responsibly, AI can be a powerful tool for improving feedback, supporting assessment, deepening understanding and preparing students for their future. Explainable AI is an important part of making that future trustworthy and aligned with human learning.