Escaping AI’s Magic Black Box
By Robert T. Pennock
Scientific explanation can help to solve the shoggoth problem.
This Article From Issue
September-October 2025
Volume 113, Number 5
Page 282
Science fiction writer Arthur C. Clarke said that any sufficiently advanced technology is indistinguishable from magic. The artificial intelligence (AI) technology of chatbots based on large language models (LLMs) is magical in just this sense. Speak a magic word to summon the AI and your wish is granted, with no explanation of how your preferred genie—ChatGPT, Claude, or Siri—does (or sometimes doesn’t quite do) the trick.
QUICK TAKE
- AI large language models (LLMs) are trained on huge datasets, but the exact programming that results from this training still remains a bit of a mystery to engineers.
- This AI black box creates ethical worries, not least because technology that is not understood cannot be repaired when it does not function as intended.
- Understanding scientific explanation can help engineers frame what questions must be answered for the use of AI to be considered on strong scientific footing.
American Scientist Comments and Discussion
To discuss our articles or comment on them, please share them and tag American Scientist on social media platforms. Here are links to our profiles on Twitter, Facebook, and LinkedIn.
If we re-share your post, we will moderate comments/discussion following our comments policy.