IMSA Independent Study: Applying System Safety Engineering and Risk Management to LLM-based applications

Safety Engineering has a history going back to the 19th century. In the past 15 years, as ML research has accelerated, those responsible for ensuring safety have become detached from this history. Over the course of this project the students learned about the history of Safety Engineering and then explored how to apply it to software systems, in particular AI-based software systems, through a hazard analysis technique called System Theoretic Process Analysis (STPA).

General Project Outline

1. Identify a system to analyze and evaluate for safety
2. Learn how to perform a basic hazard analysis using STPA
3. Define “losses” and determine what the group considered “acceptable levels” of risk
4. Identify the hazard states that can lead to a loss
5. Identify and then create the system constraints required to manage or mitigate the hazards
6. Document all of the logic, rationale, and work
7. Define tests to evaluate the efficacy of constraints built to manage the hazards
8. Perform testing based in several research-backed methods
9. Review the findings and present whether or not the group feels the project is “safe enough” for public use

Learning Outcomes

• Gain exposure to system safety engineering and understand how it relates to AI and software
• Learn the basics of at least one method for performing hazard analysis
• Explore some of the key concerns for testing AI systems and strategies for dealing with tradeoffs
• Experience the difficulty of creating consensus on what is “safe enough” and how complex it can be to attempt to make technology safe

Deliverables

• IMSA | Math Helper App | Product Brief
• Math Helper App on Hugging Face

Timeline

The independent project started in September 2024, with three students signing up to take on the work over the course of a year. The first semester focused primarily on introducing them to the critical concepts they’d need to do the work. From January 2025 to May 2025, the students worked on building tests to assess the app they designed and to generate a report on how ready it might be for public use. They presented at the IMSAloquium on April 17, 2025 as the culmination of their work.

Project Details

Over the course of the 2024-2025 academic school year I taught three high school students from the Illinois Math and Science Academy–the high school I attended–how to approach testing LLM-based applications. The goals, as stated above, were largely to introduce them to the broad context of System Safety Engineering and help them develop intuition for how to think critically about performance within generative AI enabled applications. In the first semester we focused on setting a foundation, introducing complex concepts in a digestible way and helping them explore the “why” behind testing. In the second semester they built tests for an application they co-developed. I guided them to use Hugging Face and Streamlit at the time because it would help them build quickly and focus on the output rather than getting bogged down in learning architecture or cloud hosting.

Hugging Face changed their pricing structure in January 2025 and functionally eliminated the ability to run small models on public spaces for free. This made using Hugging Face an additional barrier to the students so I had to switch to them self-hosting the application through Ollama or using Google Collab. It was an excellent demonstration on how reliance on a single vendor can derail a project. It also demonstrated how critical thinking and a willingness to adapt can overcome those barriers.

Readings

• Trustworthy Machine Learning, Chapter 3, Kush R. Varshney
• NASA Risk Management Handbook
• Engineering a Safer World, Chapter 3, Nancy Levenson
• Moral Crumple Zones Cautionary Tales in Human-Robot Interaction, Madeleine Clare Elish
• Beyond Accuracy: Behavioral Testing of NLP Models with CheckList - ACL Anthology, Marco Tulio Ribeiro
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, Apple Machine Learning Research
• Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
• Evaluation metrics and statistical tests for machine learning

Artifacts

• Testing Tutorials on Github

In Closing

I feel great pride for the students who stepped up to take this project on. They demonstrated a willingness to explore complex concepts with no hard or definitive answers. I would happily teach another group in the future. More than anything, it exposed me to the difficulty of preparing engaging lessons tailored to engage younger people in critical thinking around AI. Many people simply want to look at the cool use cases of AI, to work on the “bleeding edge”, but the three students I worked with wanted to go further. They wanted to see how to make technology safe for public use. I am grateful for getting to support them on their journey.