It has been about two months since I pushed out the previous update to Lighthouse, a code coverage plugin for IDA Pro. Overdue for an update, I spent the past several days adding new features to the plugin, addressing feedback, and fixing up bugs that have accumulated since the v0.6 release.
This post marks the release of Lighthouse v0.7 - Big ticket changes include Frida support, a right click context menu, improved usability for large IDB’s, C++ name demangling and a multitude of other tweaks + bugfixes.
On Twitter he asked for more Frida-proficient eyes to review his contribution and @oleavr, the creator of Frida, stopped by to offer his personal expertise on the pull request. Can’t ask for better support than that!
Frida is scriptable instrumentation framework that is particularly well known for its support of mobile platforms such as iOS, Android, and QNX. This is a quality addition to Lighthouse’s existing support for DynamoRIO and Intel Pin.
Support for Frida will be marked as experimental while the community helps us shake out any early bugs.
The table in the Coverage Overview now comes with a right click context menu. I have populated the context menu with a few basic amenities to enable quick user interaction with the table entries.
Function prefixing was added to this menu as a request from the community. It operates in a similar fashion as the standalone IDA prefix plugin, but is only accessible through Lighthouse’s Coverage Overview table.
Please try out the context menu when you get a chance and let me know what other actions you might like to see added to it in the future. Community requests help me keep the pulse on what is important.
Large IDB Support
While using Lighthouse with some rather large IDB’s (250k+ functions) I encountered a few annoyances that degraded its usability at these extremes. This release promises improved support for big databases.
When loading coverage data from disk, Lighthouse builds a cache of the IDB metadata in memory so that it can operate in a performant & asynchronous manner when the user begins to explore their coverage.
Previously, the metadata cache would get rebuilt every time a selection of coverage files were loaded by the user. This was largely unnecessary since the only reason to refresh metadata is when function structure gets modified, or functions have been defined/undefined in the database.
Lighthouse will now only build the metadata cache the first time you load coverage files. If a refresh is ever needed, one can use the ‘Full refresh’ option from the new right click context menu in the Coverage Overview table.
Additionally, I have given the shell’s search command a bit of breathing room when operating on a large IDB. It will now wait for one second after your last keystroke before evaluating the search request.
There have been many tweaks made throughout this release. The following is a selection of the minor changes that you might find interesting or valuable going forward.
The special aggregate coverage set now displays a red ‘X’ in the coverage combobox. This can now be used to clear all loaded coverage from Lighthouse in two easy clicks.
Coverage Color Change
A few users using the default IDA theme raised the point that the light green coverage paint was throwing off their feng shui when paired up against the blue/red coverage table. The use of green was an artifact of an older theme.
The light green paint has now been switched to a light blue for users using a light disassembly theme.
Jump Case Sensitivity
A minor quality of life / usability tweak has been made to the shell’s jump functionality. It is now case insensitive for functions starting with the reserved
Fuzzier Module Name Detection
When loading coverage data from disk, Lighthouse automatically extracts coverage from the log based on the name of the executable that was used to generate the IDA database.
In this release I have modified the coverage loader’s name matching ‘fuzziness’ to ignore file extensions as a fallback, focusing only on correlating module names between the log and the IDB. This frequently tripped up Lighthouse when loading coverage collected against Linux libraries (among other things).
In the future I plan to add a dialog for the user to manually select coverage from the log as a final fallback.
With every release of Lighthouse I generally find and fix a number of bugs. This section highlights some of the important bugfixes that will be shipping with this release.
Propagating Function Renames
Previously, the metadata cache would not get updated when functions were renamed in the database. This issue has been on the backburner for awhile, but last week MJavad opened an issue for it which prompted me to finally fix it.
Incorrect Coverage Percentages
Lighthouse has been incorrectly computing the database-wide coverage percentages. It has also been incorrectly computing the coverage percentage when using the
/ (search) command on the composing shell.
The percentages in the locations indicated above should now be closer to reality in v0.7. Existing coverage files will probably appear to skew lower (on average) in v0.7 than they did in v0.6 because of this bug.
Negative Complexities Scores
A new feature of IDA 7 is its ability to parse exception handling information (when present) from the binary. Exception handlers are an implementation specific concept dependent on abstract control flow that is only realized at runtime.
As a result, exception handlers are hard to represent statically in an intuitive manner. IDA 7 presents them as disembodied chunks in the original function’s flowchart (graph view).
Without good heuristics to compute the ephemeral edges to and from these blocks, they disrupted Lighthouse’s cyclomatic complexity approximations and produced comically wrong results for some of these functions.
For the time being, I have chosen to revert to IDA 6.X behavior by simply ignoring these chunks until I have more time to devise a better solution for representing their metrics. This means that these chunks will not be visible to Lighthouse’s painting or coverage capabilities.
In the meantime, HexRays has stated that they are continuing work in this area and will integrate exception handling information into the decompiler(s) at some point in the near future.
- Fixed a bug that made the pintool unusable (?) for 32bit binaries
- Fixed a regression that caused pseudocode painting to fail on some builds of IDA 6.8
- Fixed a race condition that could crash Lighthouse on rare occasions when saving compositions
- Fixed a bug that prevented compositions from being saved if bc95 was disabled on IDA 7 (non-default)
In the Wild
Since the last release, I have been forwarded a myriad of positive comments about Lighthouse. Among the messages have been links to a few new public works making use of Lighthouse, in the wild.
Microsoft Office RCE, by Embedi
In mid November Embedi highlighted Lighthouse in their blogpost detailing a vulnerability they discovered and reported to Microsoft. Using Lighthouse, they were able to diff coverage sets between two separate runs of Microsoft Office to quickly identify interesting (and buggy) parsing routines.
The time has come for Microsoft to show their "Skeleton in the closet". MS Office RCE 17 y.o. vulnerability you didn’t know about https://t.co/8dEnpLV4kK— Embedi (@_embedi_) November 14, 2017
The Art of Fuzzing, by Rene Freingruber
Here are my slides from my talk "The Art of Fuzzing" with the demos (workflow with AFL&WinAFL, Taint Analysis, Reversing Tricks for Fuzzing, in-memory fuzzing, DynamoRio and much more!) #heisedevsec #defcamp #defcamp2017 #itsecx #bsidesviennahttps://t.co/A0ed6iFVwn— Rene Freingruber (@ReneFreingruber) November 21, 2017
Lighthouse makes an appearance in the eighth video to diff coverage and identify interesting code (again!)
This was a well rounded release where the features, tweaks, and bugfixes all carried an equal degree of importance towards building a better user experience. The project continues to thrive off the support of the community with a growing number of researchers integrating Lighthouse into their daily workflow.
I welcome external contributions, issues, and feature requests.