October 2, 2026
Building a Human-Guided, LLM-Assisted Security Investigation System
🛡️Our alerts were doing their job: they told us when something looked unusual. The problem was everything that came next. An engineer still had to gather the evidence, work out what it meant, and write up the findings. We built an LLM-assisted workflow to handle that preparation. The investigation and every decision remain in human hands.
| Challenge | Approach | Results |
|---|---|---|
|
|
|
đź”’The diagrams and examples in this post are intentionally high level. They explain how we approached the problem without exposing source code, internal alert rules, vendors, infrastructure, credentials, or personal data.
The problem we kept running into
Whenever one of our suspicious-traffic alerts fired, an engineer had to stop what they were doing and start digging. By “suspicious traffic,” we mean unusual platform activity worth investigating, not confirmed abuse.
The engineer would open several tools, line up timestamps, look for related activity, decide which signals mattered, and write down what they found. The alert arrived quickly. The useful explanation did not.
Monitoring could tell us that something had changed, but an engineer still had to answer the harder questions:
- What happened during the period covered by the alert?
- Is this isolated or part of a larger pattern?
- Which evidence supports the alert?
- Could there be a legitimate explanation?
- Are we missing anything important?
- Does anyone need to act?
After seeing that work repeat, we asked whether the investigation could be prepared before an engineer opened the alert. That became the goal: gather the approved evidence, organize it, use a large language model (LLM) to make it readable, and put it in front of an engineer.
We wanted the alert to arrive with a head start on the investigation.
The report would not be a verdict or an automatic action. It would give the engineer the evidence, the open questions, and a better place to begin.
Why the obvious solution was not good enough
The quick version was easy to picture: send the raw alert data to an LLM and post whatever came back. That would have made a good demo. We would not have been comfortable relying on it.
The details that took real engineering work were less flashy:
- Limiting the information the workflow could retrieve
- Turning inconsistent inputs into evidence we could test
- Keeping missing evidence visibly different from negative evidence
- Letting engineers trace important statements back to their source
- Handling partial failures without covering them up
- Making sure generated text could never count as approval
The project brought together security thinking, platform engineering, operational experience, and a lot of attention to how an engineer would actually use the report.
We were not trying to bolt an LLM onto an alert. We were trying to improve an investigation workflow the team already understood.
We started with one question: what can the LLM do?
Before choosing a model, we agreed on its job.
The workflow could:
- Retrieve only the information it was allowed to use
- Put that evidence into a consistent format
- Ask the LLM to explain the supplied evidence in plain language
- Show missing or conflicting information
- Deliver the report where the team already worked
It could not make the final decision or independently take an action that might affect a member. At SoLo, we use “member” to mean someone who uses our marketplace.
That decision kept the design understandable. Gathering evidence, writing the explanation, reviewing the result, and taking action are separate steps. We can test them separately, and a failure in one step does not give another step more authority.
- The rule we kept coming back to: The LLM explains the evidence it receives. An engineer decides what that evidence means and what happens next.
How it works
At a high level, the flow is straightforward: an alert starts the investigation, the workflow gathers permitted information, the LLM prepares a report, and an engineer reviews it.
Figure 1: The main responsibilities in the workflow. This is not a map of SoLo’s internal infrastructure.
The alert starts the process, but it does not determine the outcome. The workflow gathers only what it is allowed to retrieve and arranges it into a consistent set of evidence. The LLM turns that evidence into a report. An engineer checks the report before deciding what to do.
If information is missing, the report says so. If the LLM is unavailable, the evidence can still reach an engineer. If action is needed, it goes through a separate process with its own permission checks.
What happens after an alert fires
Figure 2: Software prepares the investigation; the engineer owns the outcome.
We placed the report beside the original alert on purpose. If engineers have to remember where to find the investigation, some of its value is already lost. Keeping the signal, evidence, and unanswered questions together makes the review easier to follow.
Clean up the evidence before asking the LLM
Alert data rarely arrives in one neat format. Timestamps, identifiers, event types, and errors can all look different. Sending that raw material straight to an LLM would ask it to decode the data and investigate it at the same time.
Instead, we organize the information first:
| Part of the evidence | What it tells the engineer |
|---|---|
| Trigger | Why the investigation started |
| Time period | Which period the report covers |
| Observed signals | It reuses existing workflows and policies |
| Related activity | It can sit in front of several or older systems |
| Data quality | What is missing, delayed, or conflicting |
| Source references | Where important observations came from |
This gives us a cleaner boundary. We can test whether evidence was collected and organized correctly without having to judge the wording of the report. The LLM receives a limited set of evidence rather than an open-ended instruction to “go investigate.”
⚖️The trade-off: Organizing the evidence took more work up front. Sending raw alert data directly to an LLM would have been faster to prototype. We chose the slower start because a consistent format is easier to test, inspect, and change safely.
It also protects an important distinction: missing evidence does not mean something did not happen.
Build a report an engineer can question
We did not want a long paragraph that simply sounded confident. We wanted something an engineer could scan, verify, and disagree with.
A useful report should answer:
- Why did this investigation start?
- What did the available evidence show?
- Which observations seem related?
- What might have a legitimate explanation?
- What information is missing or unclear?
- What should the engineer check next?
Figure 3: A fictional report layout showing evidence, uncertainty, a suggested check, and action status.
This is a made-up example, not a production alert. It leaves out internal alert rules, system names, personal data, and response logic.
The report is useful because it gives the engineer facts, caveats, and somewhere sensible to continue. It does not claim to have solved the case.
What we leave to people
The LLM is good at turning dense evidence into a readable explanation. That saves time. It does not make the explanation true by default.
| The LLM helps with | The engineer remains responsible for |
|---|---|
| Summarizing supplied evidence | Why the investigation started |
| Organizing observations | Which period the report covers |
| Pointing out gaps | It reuses existing workflows and policies |
| Suggesting what to verify | It can sit in front of several or older systems |
We do not ask engineers to trust the report because it sounds certain. We give them evidence they can inspect and make gaps obvious when the investigation is incomplete.
What happens when things go wrong
Most operational failures are ordinary: a source is late, an identifier cannot be resolved, or two signals disagree.
⚠️When part of the workflow fails, we show the failure and ask an engineer to review what is available. The system does not get more freedom to act.
An incomplete report says it is incomplete. If the LLM cannot produce an explanation, the underlying evidence can still reach an engineer. And a confident sentence never becomes an approval.
“Unable to complete the analysis” is a perfectly acceptable result. It is better than making something up.
How we know whether it is useful
We are not optimizing for impressive prose. We care whether the workflow helps an engineer get to a sound decision with less repetitive work.
These are the questions that matter to us:
- Does the report arrive in time to help?
- Does it include the evidence we expect?
- Can engineers trace important statements to their source?
- Are missing and conflicting inputs easy to spot?
- Does the report reduce manual investigation work?
- Can we identify the person who approved any action affecting a member?
We are not publishing an impact number until we have a validated result that is appropriate to share externally. We are also keeping the rules that trigger alerts private. An LLM-assisted security system should be judged on whether it is accurate, useful, and safe. Beautiful writing is not the goal.
What changed for the team
The biggest difference is where the investigation begins.
| Before | AfterAfter |
|---|---|
| An alert started a manual investigation | An alert starts evidence collection |
| Engineers searched across several tools | Relevant evidence arrives in one report |
| The engineer rebuilt the timeline | The report provides a consistent starting point |
| Findings varied by reviewer | Reports follow the same basic structure |
| The next step required more coordination | Review and approval happen beside the evidence |
Our engineers still make the judgment call. They now spend less time assembling the raw material and more time deciding what it means.
The work also gave us a pattern we can reuse: give the LLM a narrow job, organize inputs before asking for an explanation, show what is unknown, and keep generated text separate from action.
That balance matters for our members. Faster investigations help us respond to real risk, while an engineer’s review helps us avoid treating one unusual signal as the whole story.
What we would do again
Start with the repetitive work
Collecting and organizing evidence is repetitive and testable. Judgment depends on the situation and can affect real people. Automating the first created value without pretending we had solved the second.
Give the LLM one clear job
“Investigate this alert” is vague. “Explain this evidence, show what is uncertain, and do not take action” is something we can evaluate.
Include missing information in the report
A clean-looking report can hide uncertainty. If something is missing, the engineer should see that immediately.
Design the review, not just the report
It is not enough to say a person is involved. The engineer needs the evidence, the caveats, a clear approval step, and a record of what happened.
Fit the workflow to the team
Putting the report beside the alert reduced tool switching and made the system easier to use. The engineer’s experience was part of the technical design.
Final takeaway
Our alerts were already doing their job. We wanted to reduce the repetitive work that began after each one fired without giving up the care that security decisions require.
The most important decision was not which LLM to use. It was deciding what we would allow the LLM to do and what would remain off limits. We collect evidence consistently, show what is unknown, make the report easy to challenge, and leave the decision with an engineer.
That is the part we are proud of, and the part we think other teams can reuse.
- The goal is not an alert that acts on its own. It is an alert that gives an engineer a better place to start.