Filesystem Scanner changelog

This is the list of version numbers of the SandboxAQ Filesystem Scanner (formerly Host Scanner). Each version number is shown with a list of changes brought by that version.

0.9.21 - 2024-10-02

Changed

• [General] When scanning an image, the image information (Docker image name, VMDK image path) is now moved to a new field in the trace without replacing the host name.

0.9.20 - 2024-09-24

Fixed

• [General] Fix performance issues on Windows when using IO limiting.
• [General] Better error reporting when --static-scanner-path is not an executable file.

0.9.19 - 2024-08-28

Fixed

• [PKCS#7] Improve error message when a Microsoft Certificate Trust List file is encountered. New log is straightforward and appears as an info.
• [JCEKS] Discard more malformed Secret Key entries in JCEKS keystores.
• [ZIP] The scanner can now process Zip archives containing offsets pointing outside the file.

Changed

• [General] -x/--exclude will now exclude paths ending with a separator (/ or \).

0.9.18 - 2024-07-30

Fixed

• [ZIP] The scanner can now process Zip archives containing empty entries marked as ‘Inflated’ (decompressed) without hanging.

Changed

• [General] -r/--root can be provided multiple times.
• [General] --scan-windows-stores argument can be be provided alongside --root or ran individually.

Added

• [General] Support CPU limiting on Windows.

0.9.17 - 2024-07-17

Fixed

• [ZIP] Close file descriptors correctly.
• [General] Allow –max-file-size and –max-zip-entry-size to be set with value 0, infinite (as documented).
• [VMDK] Fix parsing issue with scan paths.

Changed

• [General] Updated CLI argument validation error messages for uniformity and clarity.

Added

• [General] Add --io-kbps-limit argument to Filesystem Scanner. This argument sets a limit on the IOPS and allows throttling file I/0 operations. Default value is no limit.

0.9.16 - 2024-06-27

Fixed

• [General] Fix CPU limiting inaccuracies on Linux.

Changed

• [Help] Apply cosmetic and consistency style changes in --help.

Added

• [General] Add Windows Certificate Stores scan support with --scan-windows-stores.
  • Scans CurrentUser and LocalMachine locations for store names. For each store name, retrieves information for all X509 encoded certificates in the store.
  • Stores with non-ASCII characters in their names will not be scanned.
  • --scan-windows-stores is mutually exclusive with --root and --image-name.
  • Paths of certificates found in Windows Certificate Stores are displayed with the CERT:\ drive notation.
• [General] The progress bar now works with Windows compatible terminals.

0.9.15 - 2024-05-30

Fixed

• [ZIP] Resolve an issue on Windows hosts that led to “Too many open files” errors during large scans of Zip files. This issue would result in missing files in the scan, as the scanner would fail to open additional files after reaching the system’s file handle limit.
• [PKCS#12] PKCS12 files containing local machine keysets attributes no longer cause the host-scanner to miss the file during the scan. This prevents potential gaps in the scanned data.

0.9.14 - 2024-04-29

Changed

• [Logging] Redirected several internal logs messages from CLI output to the trace.
• [Logging] Add path of the file in the internal trace logs.
• [General] In case of paths both explicitly included (with --root) and explicitly excluded (with --exclude=EXCLUDE), force priority to the exclusions.
• [General] Performance improvements on file I/O.

Added

• [General] Introduce --max-file-count command line option in order to provide an optional limit to the number of files and directories the host-scanner will parse. Default value is 50 Million. 0 is considered no limit.

Fixed

• [Security] Require the value of --max-file-size to be strictly positive.
• [Security] Provide protection against TOCTOU attacks by updating the internal IO mechanism.
• [Security] Discard malformed Secret Key entries in JCEKS keystores.

0.9.13 - 2024-04-08

Added

• [PBE] Decrypt PBE-encrypted files in ZIP files with the provided passwords.

Fixed

• [Certificates] Fix a potential duplication of certificates in the database due to the presence of OpenSSL’s Trust Settings on a certificate.
• [Keystores] Improve error messages for JKS and JCEKS keystores with negative counts and malformed entries.
• [Certificates] Add missing OID translations to X.509 certificates parsing.
• [ZIP] Discard archives over 1MB uncompressed, protecting against both malicious and non-malicious files.

0.9.12 - 2024-01-30

Changed

• [General] Remove short -sc option from the CLI. Keep only the long --static-scanner-path.
• [General] Allow --password option to be repeated. PBE on-the-fly decryption is attempted with each password sequentially.
• [General] Check if trace file already exists and interactively ask the user for permission to overwrite if needed.
  • Check if the Host Scanner is ran interactively to skip permission.
  • Introduce --allow-overwrite CLI option to skip permission.

Added

• [SST] Add support for SST (Microsoft Serialized Certificate Stores).
• [SSH] Add support for SSH’s known_hosts-like files as SSH public key files.

Fixed

• [SSH] Parse all and extract all lines of files containing SSH material.

0.9.11 - 2023-07-18

Changed

• [General] Ignore directory symbolic links and junctions on Windows. This is similar to the behaviour on Linux where such equivalent symlinks have always been ignored.
• [General] Always write paths as absolute paths in trace files.

Fixed

• [ZIP] Improve error message when a zip64 or a file with any unknown compression method is encountered. New log is straightforward and appears as a warning.
• [General] Improve error message occuring when the Host Scanner is asked to write the output trace file in a directory that does not exist. The Host Scanner now checks if the output directory already exists, and displays a “cannot create trace file” if this is not the case.
• [General] Improve error message occuring when the Host Scanner is asked to write the output trace file in a directory with insufficient rights, or to overwrite an existing file without any sufficient rights. The Host Scanner now checks if the output directory has execution and write accesses, and displays a “permission denied” if this is not the case.

0.9.10 - 2023-05-22

Changed

• [PKCS#12] Improve support for Windows generated PKCS#12 algorithm identifiers.

Added

• [PKCS#5] Add support for pbkdf2 with AES PBE encryption with --password argument.
• [General] Log messages meant for debugging the parsing and analysis of files are now written to the output scan file instead of standard output/error. Other logs, such as operational logs, are written to both outputs.

Fixed

• [General] When the Host Scanner doesn’t have permission to write at the path of the output file, display an error message instead of crashing.

0.9.9 - 2023-05-15

Changed

• [General] --password argument does not appear by default in trace arguments. Use --allow-secrets-in-trace to write it again in the trace.

Added

• [General] Add --tag argument to Host Scanner. Can be specified multiple times. The provided tags are written in the trace header.
• [General] Add --allow-secrets-in-trace argument to Host Scanner. This argument will allow --password argument to be written in trace.
• [General] Add --max-files-per-second argument to Host Scanner. This argument sets a limit on the number of files scanned per second. 0 is considered as no limit and is the default value.”
• [General] Add --work-load argument to Host Scanner. This argument sets a limit, as a percentage, on the CPU load of the Host Scanner during its execution. Default value is 100% (no limit).
• [General] Add --exclude argument to Host Scanner. Can be specified multiple times. The provided files or directories are not scanned.
• [zip] Add zip module and basic parsing of zip files.

0.9.8 - 2023-03-28

Added

• [PKCS#12] Add support for PKCS#12 ShroudedKeybags in keystore parsing.

0.9.7 - 2023-01-04

Changed

• [PKCS#12] Remove unnecessary warnings about PKCS#12 parsing.
• [Certificates] Add missing algorithm identifier for md2withrsa signature.
• [General] Add alternative algorithm identifier for DSA keys.
• [General] Improve reporting of unknown algorithm identifiers.

Added

• [PPK] Detect PPK files (used by Putty) and extract their content.
• [PKCS#7] Detect PKCS#7 files and extract their content.
• [SSH] Add the parsing of the public keys coming from SSH certificates.

0.9.6 - 2022-09-15

Changed

• [PEM] Extract all certificates from concatenated PEM files.

Added

• [General] Added the --password option to try the decryption of PBE encrypted informations.
• [PKCS#12] Extract content of Data items.
• [PKCS#12] Extract content of EncryptedData items if --password is provided and decryption succeeds.
  • Support of PBES2 encryption with pbkdf2 as KDF, AES-(128/192/256)-(ECB/CBC) and 3DES-CBC as encryption schemes.
  • Other algorithms are not decrypted but the algorithm names are exported for analysis.

0.9.5 - 2022-06-28

Changed

• Deprecate the --include-keys option. Keys are now included by default.
• Detect PEM blocks preceded by some plain text, up to 32 kiB after the beginning of the file. Previously PEM content was only detected at the beginning of a file.

Added

• Detect JCEKS keystores and extract their content, if unencrypted.
• Add support for Windows.

Fixed

• Fix crash on certificates containing invalid UTF-8 strings inside distinguished names.
• Fix crash on some malformed PGP keys.
• Increase robustness of scanner against parsing bugs by making it recover and continue scanning.
• Fix a situation where some combinations of command-line options caused the same parser modules to be applied to files more than once.
• Ensure correct JSON encoding of the trace header (e.g. correct escaping of \ characters in Windows file paths).
• Fix crash on keystore private key entries with empty certificate chains.
• Fix failure to detect PEM files with Windows CRLF line endings.
• Fix crash on some special files where true and reported file lengths differ (e.g. in /sys).

0.9.4 - 2022-04-08

Changed

• Change the --max-file-size option to interpret 0 as “no size limit”. Any strictly positive value is interpreted the same way as before (file cutoff in bytes).

Fixed

• Improve encoding of certificate serial numbers so that such certificates have a better chance of being correctly matched with certificates coming from other sources such as the Java Tracer.
• Improve performance of scanning:
  • Improve detection of ASN.1 files to reduce memory usage and time spent in scans.
  • Exclude CPIO and system map files from scan.

0.9.3 - 2022-03-02

Added

• Add option --max-file-size to change the size limit for scanned files. The default value is 1 MB.

Changed

• Limit the amount of data read from any single file to 1 MB by default. This avoids performance issues when large files are mistaken for ASN.1 files.

Fixed

• Prevent the Host Scanner from missing some files when /sys is in the scope of the scan, even if those files are outside of /sys.

0.9.2 - 2022-02-01

Added

• Add initial support for PKCS#12 keystores

Changed

• Fix host name for Docker image scans: it now reports the image name.

0.9.1 - 2021-10-29

Fixed

• Fix container image file locations to not contain the temporary prefix used for scanning (e.g. /etc/ssl/cert.pem instead of /tmp/tmp_name/etc/ssl/cert.pem).

0.9.0 - 2021-10-13

Added

• Add initial support for PGP keys.

0.8.0 - 2021-08-04

Fixed

• Catch and handle “end of file” errors coming from the Java static scanner if it is used in combination with the Host Scanner.

0.7.1 - 2021-07-29

Added

• Add on-the-fly gzip compression of scans.

Fixed

• Fix stack overflow when scanning files with a large number of line breaks.
• Improve error message when the Host Scanner is used in combination with the Java static scanner.

Changed

• Rename CLI option --static_scanner_path to --static-scanner-path
• Rename CLI option --image_name to --image-name

0.7.0 - 2021-07-09

Added

• Add Docker image scanning.

0.6.0 - 2021-06-11

Added

• Add support for EC and DSA private keys and certificates.
• Add support for DER encoding.
• Include encoded certificates in the scan file.

0.5.0 - 2021-04-16

Added

• Add hard-coded strings parsing of JAR files.

0.4.0 - 2021-02-10

Added

• Add parsing of JKS keystores.

0.3.0 - 2020-04-27

Added

• Add a progress bar showing the number of files scanned.

Changed

• Change log level of “Unknown OID” message from warning to debug.

Fixed

• Improve performance on large directory trees: less RAM usage, less computing and fewer syscalls.
• Fix file being counted twice, and potential infinite loop, by not following symbolic links.
• Fix freeze on files without an end, such as some special files in /sys/kernel.

0.2.0 - 2020-04-16

Changed

• Make Host Scanner a lot less verbose by default.

Fixed

• Fix stack overflow when scanning large directories.

Added

• Add --verbose and --quiet options to control the verbosity.

0.1.0 - 2020-04-07

Initial release.