IVRE: new (cool) features

Long time no post… so here is a Prévert-style inventory of some recent IVRE’s features.

Top ports for a given service

It has been possible for a long time now to query the most common services on a given port (by querying the top values for the pseudo-field service:80 or probedservice:80).

It is now possible to query the most used ports to offer a given service, by querying the top values for the pseudo-field port:[service name]. Of course, the common ports (e.g., 22 for ssh) are not interesting here, but the uncommon values might be. This can be done from both the Web interface and from the Python API.

As an example, here is the output of the top values for port:http within the Modbus scan, from the Python API:

>>> from ivre.db import db
>>> from pprint import pprint
>>> pprint(list(db.nmap.topvalues('port:http')))
[{u'_id': 80, u'count': 6246},
 {u'_id': 443, u'count': 860},
 {u'_id': 8080, u'count': 790},
 {u'_id': 8088, u'count': 321},
 {u'_id': 81, u'count': 202},
 {u'_id': 8081, u'count': 136},
 {u'_id': 5080, u'count': 98},
 {u'_id': 8008, u'count': 98},
 {u'_id': 10000, u'count': 89},
 {u'_id': 8443, u'count': 79}]

And here is how top values for port:ssh looks like from the Web interface:

../../../_images/ivre-top-port-http.png

CPE support

CPEs reported by Nmap are now stored in IVRE database, and displayed in the WebUI. They can also be used to query the database, with the cpe: filter (WebUI) and the .searchcpe() method (Python API).

This feature is an idea of and has been implemented by @fmonjalet.

../../../_images/ivre-CPE.png

Merge scan results

This feature has been requested by two pentesters who usually start their network discovery, particularly when the address space is important, with some Nmap scans of a reduced set of ports (for example, they run nmap -A -sS -p 80,443 TARGETS, then -p 25,110,143, etc.).

Since more comprehensive Nmap scans can last for a really long time, splitting the scan in “sub-scans” makes it possible to start analyzing the results as soon as the first “sub-scan” is over, while the next ones are still running.

To use this approach in IVRE, it is now possible to merge host documents (using the method ivre.db.DBNmap.merge_host()), and this can be used to merge a result with current results in the database, using nmap2db with the option --merge.

OCR against screenshots

Following an idea from Henri Doreau, the screenshots are now analyzed by Tesseract and the found “words” are stored in the database, indexed.

They can be used as filters and as a field for top values:

>>> from ivre.db import db
>>> import re
>>> admin = re.compile('admin', re.I)
>>> db.nmap.get(db.nmap.searchscreenshot(words=admin)).count()
205

This is still an experimental feature, and I’m not sure yet whether it will prove useful or not.

Count open ports

Thanks to Xavier Martin, it is now possible to filter results based on their number of open ports, by specifying either a number or a range.

With the CLI tool scancli:

$ scancli --count --countports 10 10
176
$ scancli --count --countports 0 10
11535
$ scancli --count --no-countports 0 10
625

With the Python API:

>>> from ivre.db import db
>>> db.nmap.get(db.nmap.searchcountopenports(minn=10, maxn=10)).count()
176
>>> db.nmap.get(db.nmap.searchcountopenports(minn=0, maxn=10)).count()
11535
>>> db.nmap.get(db.nmap.searchcountopenports(maxn=10)).count() # better
11535
>>> db.nmap.get(db.nmap.searchcountopenports(maxn=10, neg=True)).count()
625
>>> sum(db.nmap.get(db.nmap.searchcountopenports(minn=0, maxn=10,
...                                              neg=neg)).count()
...     for neg in (False, True))
12160
>>> _ == db.nmap.get(db.nmap.flt_empty).count()
True

Web interface code

@commial is doing a huge work under the hood, especially around the Web interface, which is a bit of a thankless job because it goes mostly unnoticed, but it is really important for the code maintainability.

Contributors

Some of these features have been implemented by new contributors, and that is a really good news! Thanks!