Empty results object in resulting JSON when running `detect-secrets scan`, but results are expected (and returned with `detect-secrets-hook`)

[bender-the-greatest]

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  detect-secrets scan is not returning any results in the returned JSON, though git ls-files -z | xargs -0 detect-secrets-hook seems to work as expected.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

1. Change to local repo directory
2. Set up virtual env for detect-secrets: py.exe -3.11 -m venv detect-secrets
3. Activate venv: ./detect-secrets/Scripts/Activate.ps1
4. Install detect-secrets if missing: pip install detect-secrets
5. Run detect-secrets scan
   • The full command we want to work is detect-secrets scan --all-files --force-use-all-plugins --exclude-files FETCH_HEAD, but this can also be demonstrated with no additional arguments.
6. The resulting JSON file contains an empty list of results, but we know there should be some items being identified in this repo.

• What is the expected behavior?
  The results object in the resulting JSON should have the expected secrets or false positives identified so it may be used to create a baseline secrets file.

• What is the motivation / use case for changing the behavior?
  We have an ADO pipeline scanning our organization's Git repositories to flag when potential secrets are found in source code. We get results flagged when it is run as part of this pipeline (pipeline OS is Ubuntu). However, trying to reproduce the same command against the same repo but on a Windows development machine yields no results returned.

Because we have a high number of false positives being identified in files that are updated somewhat frequently (e.g. file hashes and registry paths are often misidentified as Base64 High Entropy String) we want to run the same command locally so we can make sure to update our secrets baseline before the pipeline scan occurs to prevent having to double back to re-confirm secrets flagged after the pipeline detect-secrets has run and we are notified at a later time.

• Please tell us about your environment:

  • detect-secrets Version: 1.5.0
  • Python Version: 3.11
  • OS Version: Windows 11 (via PowerShell 5.1)
  • File type (if applicable): git repo (mostly ruby files and powershell scripts)

• Other information
  Installing findutils on Windows and using git ls-files -z | xargs -0 detect-baseline-hook does flag the items we are expecting to see, but this is both different than what is returned with the detect-secrets scan entrypoint and is also not suitable for creating a baseline secrets file.

[gpflaum]

@bender-the-greatest Do your files contain Unicode (non-ASCII) characters? It might be the problem reported in #641.

I'm seeing the same thing: A scan on Linux finds secrets. A scan on Windows does not. If I convert the UTF-8 file to Windows 1252 code page, a scan on Windows finds the secrets.

[gpflaum]

Here's a file which demonstrates the problem. A scan on Windows doesn't report a potential secret.

Save as utf-8 right curly double quote.js:

({
    PASSWORD: "test",
    // U+201D
    RIGHT_CURLY_DOUBLE_QUOTE : "”"
})

Surprisingly, a scan on Windows does find the secret in this file: utf-8 left curly double quote.js:

({
    PASSWORD: "test",
    // U+201C
    LEFT_CURLY_DOUBLE_QUOTE : "“"
})