Reverse Engineering with AI: Give the Agent a Way to Be Wrong

Contents

“What does this function do?” can take you through half a binary before you have an answer. I was doing that work long before coding agents existed, and I still enjoy following the clues. With an agent, though, I can spend less time chasing each one myself. It can investigate a call, suggest a structure, or recognize a compiler idiom while I work on the larger question. That change in pace made me want to try more ambitious projects.

Lately, I have also been seeing more game reverse-engineering projects, ports running in browsers, and discussion of REA, or Reverse Engineer Anything . Seeing those results made me wonder how other people were working. Was there something in their setup that I was missing, or could I get further with the tools I already had? I wanted to understand their process, but above all I wanted to learn something new, rather than just do familiar work faster.

I tested REA, but it added little to my setup. I already had skills, tools, and scripts for the common tasks, so much of what it offered was already covered. It can be a useful starting point if you do not have that setup yet. Some of the discussion makes it sound as though reverse engineering with agents depends on REA, but agents can use disassemblers, debuggers, and scripts directly. For me, the more useful question was what to let the agent do with those tools.

Reading assembly was already part of the answer. An agent can often work with a binary as another representation of a program: inspect the instructions, propose a C implementation, and explain the relationship between the two. Its knowledge of compiler idioms, libraries, and data structures gives it places to look, including patterns I might not recognize myself. That makes it useful for more than translating one instruction at a time.

The difficulty is deciding whether the proposed C actually describes the binary. Compilation can erase names, types, and source structure; optimization can make different source programs look alike. The agent’s output is a possible explanation, not necessarily the right one, but having a hypothesis to test is better than having nothing to go on. To make sustained progress, I wanted the agent to be able to test that hypothesis without waiting for me to inspect every attempt.

That brought me back to matching decompilation, a technique I had known about for years but had never tried because of the time it demanded. Alongside clean-room reimplementation, I have always found it one of the most elegant forms of reverse engineering: reconstruct source, compile it with the original toolchain, and recover the same machine code. Now it also gave me something new to learn and a concrete way for the agent to check its work. Differences could point to a wrong type, a different expression, or a missing piece of build context, giving it feedback to revise its own candidate.

This was the setup I wanted to try: give the agent the binary, the compiler, and a comparison it could run itself. It would still need to read the assembly and work out a plausible implementation, but it would also have a way to check its progress.

Over the past few months, I have been studying the Geely EX2 and its electronics in detail. Recently, I managed to dump the storage from its instrument cluster, or IPK. Among its contents were the Linux executables responsible for drawing the interface, handling audio, and running the cluster’s other application functions.

I plan to cover the IPK itself in more detail in another article, but these binaries gave me a good target for learning matching. The work could also help with two things I wanted to explore afterward: making visual changes to the cluster software and building a WebAssembly tech demo of its interface.

With that in mind, I put together a plan: try matching with agents and as little manual intervention as possible. Getting there still took time. I had to investigate the toolchain, prepare the scripts, and build the feedback loop. The idea was that, once those pieces were in place, the agents could inspect a function, write a candidate, compile it, and use the comparison results to keep working without asking me to check every attempt.

The work moved quickly. But the important part was not how fast an agent could write C or explain ARM instructions. It was how much progress it could make without waiting for me to judge each attempt and how reliably the tools could stop a convincing mistake from becoming an accepted result.

An unusually helpful binary #

The starting point was six final ARM executables from the application partition. Not the original source tree, not the original relocatable object files, and not a complete build environment.

There was, however, a considerable advantage: the executables retained symbols and DWARF debugging information. The application compilation units recorded -O0 -g3.

DWARF supplied function signatures, local variable names and types, structure layouts, compilation-unit boundaries, stack locations, and source line information. At -g3, macro records also preserved includes and macro definitions. The binary was still missing its source, but it had kept a rather detailed description of where it had last seen it.

That changed the agent’s job. Instead of inventing names and plausible types, it could recover constraints and propose source that satisfied them. Disassembly remained the authority for the instructions; DWARF constrained the source-level explanation.

The compiler identity mattered just as much. Five executables used Linaro GCC 7.3.1; the remaining one used Linaro GCC 7.4.1. Matching required reproducing the compilation settings too, including the instruction set and floating-point ABI. The application code was built with -O0 -g3: optimizations disabled and extensive debugging information retained. That gave me both more source-level evidence and generated code that was sensitive to how the C was written.

I initially ran the original Linux toolchains in containers on my ARM Mac, which meant running the x86-64 compiler binaries through translation or emulation. Later, I adapted the setup to run under WSL2 on a faster Windows machine to see how much that would help. The biggest speed gains, though, came from improving the setup and scripts: reducing repeated work and making the validation process more efficient.

Turn reconstruction into an executable question #

The useful unit of work was a function, with its real compilation-unit context. An agent needed to inspect the reference, write a candidate, compile it, and see precisely which constraint it had violated.

The project CLI separated those operations. Here, $TARGET selects the executable being reconstructed:

sh
# Inspect the reference's types, locals, lines, and code references.
.venv/bin/python scripts/ipk.py unit "$TARGET" list.c \
  --function list_init

# Allocate scratch space, then compile a candidate with the target toolchain.
export IPK_RE_WORKDIR="$(.venv/bin/python scripts/ipk.py workspace)"
.venv/bin/python scripts/ipk.py compile "$TARGET" \
  "src/$TARGET/utility/list.c" --output "$IPK_RE_WORKDIR/list.o"

# Compare one function in the candidate object against the final ELF.
.venv/bin/python scripts/ipk.py compare "$TARGET" \
  list_init "$IPK_RE_WORKDIR/list.o"

The idea was to give the agent a setup it could use on its own. It could inspect the binary, compile a candidate, read the comparison, and try again without needing me to run commands or interpret each result.

I have seen the best results with agents when this feedback loop is clear, not just in reverse engineering. The agent needs to know what it is trying to achieve and be able to inspect whether its work is getting there. When the goal is vague or the result is hard to check, it becomes much harder to decide what to try next. People run into the same problem.

A failed attempt could mean different instructions, a wrong call target, an unresolved relocation, a different stack slot, or a mismatched type. Those are different problems. Returning a single similarity score would have discarded much of the information needed to fix them.

A tiny linked-list function illustrates why this matters. The reconstructed list_is_empty retains this shape:

c
bool list_is_empty(const list_t *list)
{
    if (list->next == list && list->next == list->prev)
    {
        return true;
    }

    return false;
}

For a well-formed circular list, the second condition is redundant. An agent writing fresh code might remove it. Even preserving both conditions and replacing the body with return a && b; was not enough: under this compiler and profile, that candidate introduced an additional and/uxtb pair absent from the reference.

The shorter source was not the matching source. No amount of confidence in the equivalence of the Boolean expressions could answer the actual question. A compile-and-compare cycle could.

The same issue appeared at a finer scale in list_init:

c
list->next = list->prev = list;

The reference stores prev, reloads it, and then stores next. At -O0, the source expression’s shape leaves visible evidence. This is a task where an apparently helpful cleanup can erase the thing being recovered.

The goal was a bug-for-bug reconstruction, as close to the original as I could verify. That meant preserving redundant checks, awkward expressions, and actual defects rather than letting the agent clean them up. I wanted to understand and reproduce what was there first. Any fixes or visual changes would come afterward, separately from the matching baseline.

Comparing an object to an executable is not a byte diff #

The candidate compiler produces a relocatable object. The reference is already linked. A direct byte comparison will therefore reject some correct candidates, while an excessively forgiving comparison can accept incorrect ones.

Consider an external function call. The candidate object contains a Thumb bl instruction with an R_ARM_THM_CALL relocation. The executable may contain a blx to an ARM entry in the procedure linkage table. The bytes differ because the linker has done its job, not necessarily because the candidate is wrong.

The comparator cannot simply ignore all call instructions. It must check that the candidate relocation names the expected function and that the reference instruction actually reaches that function, or the PLT entry associated with its dynamic symbol. Only then can that particular call site be excluded from the ordinary byte comparison.

Addresses loaded through movw/movt pairs need similar treatment. A plausible-looking address is not enough: the comparator checks the corresponding instructions, destination register, symbol or allowed constant content, and addend against the linked reference. Defined data objects also require checks on their size and initial contents.

This produced deliberately specific verdicts, including:

  • FUNCTION_BYTES_EQUAL_ONLY for equal function ranges under the local checks;
  • FUNCTION_BYTES_EQUAL_CALLS_RESOLVED when call relocations have been checked against their linked destinations;
  • FUNCTION_BYTES_EQUAL_RELOCS_RESOLVED when supported data references or other relocation forms have also been resolved.

“Not comparable” remained a valid result. If the comparator could not resolve a relocation, the function remained unverified. I had to understand the reference before deciding whether the difference was expected.

That is an important constraint when the agent can edit both the candidate and its validator. The task is to make the source satisfy the evidence, not make the evidence easier to satisfy. New acceptance rules need negative cases: a nearby wrong symbol, a wrong addend, or a different initializer must still fail.

Although I wanted the agents to work with as little intervention as possible, I still had to step in throughout the process. Some adjustments made the checks faster, as I mentioned earlier; others were needed to make the validation more reliable.

The comparator, for example, could report a match when the candidate used a string literal where the original referenced a writable table. It also accepted a static variable with the wrong initial value. Resolving the address was not enough: the checks needed to account for what it pointed to as well. I tightened those checks and tested deliberately incorrect candidates to make sure they failed.

An automated verdict is only useful if the automation has earned the right to issue it. Otherwise, the agent can become very productive at collecting false positives.

Equal instructions were only one gate #

Different source declarations can generate the same machine code. At -O0, changing a local’s name may leave .text untouched. So can some changes in signedness or prototype spelling.

The project therefore also compared the candidate’s DWARF interface: return and parameter types, local variables, linkage, lexical blocks, stack locations, and the structure of relevant types. The line table supplied another independent constraint on how instructions were divided among source lines.

This helped narrow down the source, but some details were still missing. The debug information preserved macro definitions, but I still had to reconstruct where they were used. In some functions, the agent used explicit #line directives because it had not found a source layout that reproduced the recorded line sequence.

Sometimes line numbers were part of the program itself. In one source file, logging macros passed __LINE__ to syslog, so moving source around changed an immediate in the generated instructions. In those places, blank lines were no longer just formatting.

Build context surfaced in other ways. __DATE__ and __TIME__ affected build strings; SOURCE_DATE_EPOCH reproduced the recorded timestamp. Assertions in the identified upstream stb_image header embedded an absolute __FILE__ path. Reconstructing the logic without reconstructing that compilation context left real differences.

The feedback loop made these details tractable because each hypothesis had a small, observable consequence. Change a type, an expression, a line boundary, or a build input; compile again; inspect what actually changed.

Here is an example from the lighting-model reconstruction:

text
model_light_E2_Get_Poll_AreaQue FUNCTION_BYTES_EQUAL_RELOCS_RESOLVED
    line rows: DIFFER ...
    interface: match (DWARF)

The machine-code verdict had become positive, and the DWARF interface matched, but the line-table requirement still failed. The function was not ready to land. A separate status gate made that distinction executable instead of leaving it as a note the agent might overlook.

Matching the bugs too #

The objective was reconstruction, including defects. Fixing a bug in the matching tree would make it a worse description of the original.

One example was a checksum diagnostic in the updater. Its logging macro expanded to two statements, but its use inside a for loop had no braces. The first statement belonged to the loop; the second ran afterward, when the index was already 16, and read beyond a 16-byte checksum array.

Another was a retry timer that converted monotonic milliseconds from double to a 32-bit unsigned long. The ARM instruction used for that conversion saturates at 0xffffffff. From approximately 49.7 days of monotonic time onward, those timer values stop growing.

Running the tests on a 64-bit host could miss this issue: unsigned long was 64 bits there, but 32 bits on the target. To check behavior under the target ABI, the setup also compiled tests with the original ARM toolchain and ran them through qemu-arm.

The checks consequently had separate jobs. Native tests exercised selected behavior. ASan and UBSan inspected the paths those tests reached. Tests compiled by the original Linaro compiler also ran under qemu-arm with the toolchain’s glibc.

The setup needed work too #

I did not get the setup right on the first attempt. Once the agents started using it, I kept finding things to adjust: checks that repeated work, results that needed more context, and tasks that were less independent than I had expected. Making the process efficient took several rounds of watching where the agents got stuck, investigating why, and changing the tools or the way I divided the work.

Some problems were outside the reconstruction itself. At one point, the compiler was reading stale source files through a container mount. In other cases, functions needed surrounding code to match correctly, so treating each one as an isolated task did not work. The agents needed a setup that accounted for those conditions.

I could prepare a starting point, but I had to run real reconstruction tasks to see what was missing. The feedback loop helped the agents improve their candidates, and watching them use it helped me improve the setup.

Fast work still had a bottleneck #

The main reconstruction work ran from November 10 to 12, 2026. I was surprised by how much I could get done in that time, especially given how slowly it started. The first functions were extremely simple, chosen to test the workflow and catch basic problems. Even with those, I spent a lot of time investigating failures and adjusting the setup.

I deliberately left the larger parts for later. I did not expect to get the setup right at the beginning, and I wanted to find its problems on small functions before giving the agents more demanding work. By the last day, the process was moving much faster. That was when I made the most progress and used the most agents in parallel.

Getting candidates to match was only part of the work. One 49-function unit spent substantial time in the validation and commit process, which led me to look for checks I could speed up or run in parallel.

The pipeline was generating an inventory, then running a separate status check that compiled and compared the same inputs again, followed by behavioral tests. With that repeated for each function commit, certification could cost more than the next reconstruction attempt. Improvements brought parallel target/unit processing and overlap between independent matching and test work.

There was a context cost as well. Early sessions gathered unit evidence through many separate queries and had reviewers rediscover it. The unit command consolidated that evidence into one response. Generated CSV inventories replaced growing prose progress logs. A landing tool replaced repeated ad hoc staging scripts.

Much of the practical speed came from removing waiting, repeated discovery, and bookkeeping from the path between a hypothesis and a trustworthy result.

What the result actually means #

The recorded inventory reached the following per-function results:

ExecutableMatched application functionsFunction bytes covered
Car_Info_RM37342,606
Cluster_Control_Process14628,402
Cluster_Display_Process668355,002
Cluster_Audio_Process234,884
system_monitor102,740
update32089,072
Total1,540522,706

Those counts cover the selected application compilation units, including incorporated library code. They exclude compiler runtime and libc units. Identifying a compatible public library version was a legitimate shortcut: recovering upstream source is better than asking an agent to reinvent a known image decoder. It still had to pass the same matching checks in the recovered compilation context.

All inventoried application functions passed the matching checks. The final executable layout and some data initializers still needed separate verification.

Give the agent an informative failure #

For the next project, I would start with the same approach: a clear goal, tools the agent can use on its own, and a way to check the results.

The reference stayed fixed. The toolchain was pinned. Inspection returned the evidence needed for a bounded task. Compilation was reproducible. Comparison explained which constraint failed. The agent could run those steps without asking me to inspect each candidate. Accepted work entered the inventory through checks rather than through an assistant’s progress report.

That also changed where my attention was useful: choosing the reconstruction boundary, checking that a new comparison rule could not accept the wrong thing, and deciding what evidence was sufficient. Reading every generated line of C was not the only way to supervise the work.

This was a favorable target, with unusually rich debug information and unoptimized application code. The work showed me how much the setup matters alongside the model itself.

A company can hire a capable person and still make their work difficult through unclear goals, poor tools, and slow feedback. I think projects using AI need the same attention to working conditions. The setup should let the agent use its strengths, such as recognizing patterns and trying implementations quickly, while accounting for its weaknesses. Here, that meant giving it access to the evidence and a way to test its ideas, without accepting a convincing explanation as a correct reconstruction.

The agent did not need to be right on the first attempt. It needed a reliable way to discover that the attempt was wrong, and enough detail to make the next one better.