Rapid7 Automates Threat Hunting with LLM-Powered Pipeline
Rapid7 has developed an automated threat hunting pipeline that leverages large language models to rapidly translate threat intelligence into actionable detection queries.
Security teams are inundated with threat intelligence reports, but translating this information into effective, behavior-based threat hunts has traditionally been a time-consuming manual process. Rapid7's Internal Security team has addressed this challenge by creating an automated threat hunting pipeline designed to significantly accelerate the conversion of threat intelligence into structured, executable hunt plans.
The pipeline's core innovation lies in its use of large language models (LLMs) to process raw threat intelligence, such as blog posts and DFIR reports. The system is engineered to extract adversary behaviors, map them precisely to MITRE ATT&CK techniques, and subsequently generate detection queries tailored for various security tools. This automation aims to drastically reduce the manual effort and time typically required, moving from days to minutes for initial hunt plan generation.
Traditionally, a single threat intelligence report detailing numerous adversary techniques could consume an analyst's entire week to fully operationalize. This manual workflow involves reading, interpreting, mapping to ATT&CK, writing and validating queries for different platforms, and then triaging results. When multiple high-value reports arrive concurrently, this manual approach quickly becomes unsustainable, leaving organizations vulnerable.
Rapid7's pipeline operates in four distinct stages. First, it ingests threat intelligence by accepting URLs or pasted text, cleaning the content to remove extraneous elements and ensure a focused input for the LLM. This stage is crucial for preventing irrelevant website data from skewing the analysis.
In the second stage, the cleaned content is fed to an LLM prompted to act as a MITRE ATT&CK analyst. The model identifies and extracts adversary techniques, providing their IDs, names, tactic categories, and a brief description of their application by the threat actor. This stage specifically focuses on offensive behaviors, deliberately excluding defensive recommendations to maintain the hunting value of the source material.
The third stage involves generating detection queries for each identified technique. The pipeline produces content for tools like Rapid7's InsightIDR (using LEQL), Velociraptor (VQL), and Sigma rules, with potential for YARA rules. Crucially, all generated queries undergo mandatory review by a human analyst to ensure accuracy, correct syntax, and operational suitability before deployment.
Finally, the fourth stage assembles a comprehensive markdown hunt plan. This plan is organized by ATT&CK tactic and includes an executive summary, an indicator sweep section, and the generated behavioral hunting queries with clear explanations. This structured output allows analysts to easily inspect, edit, execute, and reuse the hunt plans.
A key feature of the pipeline is its persistent query cache. Generated queries for each technique are saved, creating a reusable library that reduces future processing costs and execution times. This cache also facilitates a feedback loop, allowing analysts to refine and improve queries over time, with those improvements being retained for subsequent hunt plans. This approach helps build an organic understanding of recurring adversary behaviors and prioritizes detection efforts on the most relevant techniques.