13.4 · Resource Overload

Explore 1 source-supported AI security cases for 13.4 · Resource Overload in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

An attacker overloads the computation, memory, or service capacity used by an AI system. Resource-intensive processing can then slow down, fail, or prevent other tasks from completing. The target is the finite capacity supporting the agent, not necessarily the accuracy of its reasoning.

Official framework source ↗ · Open interactive explorer ↗

Validated Research · · Practical overlap

Achieving Code Execution in MathGPT via Prompt Injection

The publicly available Streamlit application MathGPT (https://mathgpt.streamlit.app/) uses GPT-3, a large language model (LLM), to answer user-generated math questions. Recent studies and experiments have shown that LLMs such as GPT-3 show poor performance when it comes to performing exact math directly. However, they can produce more accurate answers when asked to generate executable code that solves the question at hand. In the MathGPT application, GPT-3 is used to convert the user's natural language question into Python code that is then executed. After computation, the executed code and the answer are displayed to the user. Some LLMs can be vulnerable to prompt injection attacks, where malicious user inputs cause the models to perform unexpected behavior. In this incident, the actor explored several prompt-override avenues, producing code that eventually led to the actor gaining access to the application host system's environment variables and the application's GPT-3 API key, as well as executing a denial of service attack. As a result, the actor could have exhausted the application's API query budget or brought down the application. After disclosing the attack vectors and their results to the MathGPT and Streamlit teams, the teams took steps to mitigate the vulnerabilities, filtering on select prompts and rotating the API key.

Why it relates: An injected compute-forever request produced non-terminating Python. The application became unresponsive and its host subsequently restarted. The observed failure consumes the code-execution capacity supporting the AI application. The shared boundary is finite execution capacity. MathGPT is an LLM-to-Python application; the source does not establish a multi-agent system or an energy-latency attack against model hardware optimizations.

The publicly available Streamlit application MathGPT (https://mathgpt.streamlit.app/) uses GPT-3, a large language model (LLM), to answer user-generated math questions. Recent studies and experiments have shown that LLMs such as GPT-3 show poor performance when it comes to performing exact math directly. However, they can produce more accurate answers when asked to generate executable code that solves the question at hand. In the MathGPT application, GPT-3 is used to convert the user's natural language question into Python code that is then executed. After computation, the executed code and the answer are displayed to the user. Some LLMs can be vulnerable to prompt injection attacks, where malicious user inputs cause the models to perform unexpected behavior. In this incident, the actor explored several prompt-override avenues, producing code that eventually led to the actor gaining access to the application host system's environment variables and the application's GPT-3 API key, as well as executing a denial of service attack. As a result, the actor could have exhausted the application's API query budget or brought down the application. After disclosing the attack vectors and their results to the MathGPT and Streamlit teams, the teams took steps to mitigate the vulnerabilities, filtering on select prompts and rotating the API key.

SecureFlow · Read the full case ↗