Feature: encourage developers to share SAST results

[laurentsimon]

Platforms like lgtm.com and sonarcloud.io enable SAST and make the results public. I think we should encourage this among projects. Given this issue #1268, we could then score the SAST check as follows:

1. If a SAST tool is enabled, give 4 points
2. If the SAST tool is enabled on PR, give additional 3 points
3. If the results are public, give an additional 3 points

[laurentsimon]

cc @evverx

[evverx]

If a SAST tool is enabled, give 4 points

I'm not sure about sonarcloud.io but LGTM is usually enabled outside GitHub and if it isn't used to analyze PRs it's hard to tell whether it's actually used by projects. I think the presence of files like https://github.com/systemd/systemd/blob/main/.lgtm.yml and https://github.com/systemd/systemd/tree/main/.lgtm/cpp-queries could be a hint but I don't think projects have to keep those files if they can be built by default of it they don't additionally use custom queries.

Coverity is another beast that can be integrated in mysterious ways that's hard to detect. In systemd and util-linux scripts sending data to it are kind of hidden in GitHub actions run once a day: https://github.com/systemd/systemd/blob/main/.github/workflows/coverity.yml

Other than that I like the idea in general. Thanks!

[evverx]

I'd add that with Coverity it isn't even clear what "public" means :-) Even when reports are "public" there has to be a Coverity account. Without it, Coverity asks people to sign in as far as I know.

[laurentsimon]

Thanks. By public, I meant that the results are available for anyone to see.

Do you know if there are public APIs to retrieve the data? For example, systemd's results seem to be available at https://lgtm.com/projects/g/systemd/systemd/?mode=list... but it's too brittle to parse html this way, so I'd rather use a REST API. I found this but I've not looked at whether this is for repo owners only.

sonarcloud has a similar public interface, e.g., https://sonarcloud.io/summary/new_code?id=phalcon_phalcon and some API https://sonarcloud.io/web_api

If you know anyone at one of these companies, feel free to cc them on this issue.

[evverx]

Last time I checked the license of LGTM explicitly forbade people from using its API if it wasn't related to sending code of open-source projects hosted on GitHub. I'm not sure if anything has changed since then though. It was acquired by GitHub at some point.

Coverity Scan isn't responsive usually so I wouldn't even try.

[evverx]

According to https://lgtm.com/tos

You may not use any automated technology to scrape, mine or gather any analysis, results or other information from the LGTM service or otherwise access the pages of the LGTM service for any unauthorized purpose.

[laurentsimon]

I think we'd be using the API only for open-source project hosted on GitHub, so we should not be in violation. cc @josepalafox

[josepalafox]

If you can check whether sast is enabled via API you're fine I believe.

[laurentsimon]

Thanks @josepalafox. Another idea is to mine the data to infer time to fix, etc., and use this info in the Vulnerabilities check #935 (comment)

[evverx]

On second thought considering that projects can be added to LGTM by virtually anybody I think it should be safe to say that if it isn't used to analyze PRs it isn't used in general.

Now that restrictions are slowly being lifted It would be great if LGTM could also be used to analyze project hosted elsewhere: https://sourceware.org/bugzilla/show_bug.cgi?id=28659#c2 :-) Seriously though, considering that scorecard can be used anywhere in various ways those terms of service would stop me anyway if by analogy with cron jobs run by analogy with https://metrics.openssf.org/ or https://deps.dev/ I wanted to set up a large-scale scorecard analysis of thousands of projects on a regular basis.

[evverx]

Looking at https://sonarcloud.io/terms.pdf, it seems sonarcloud forbids automated applications as well

An
Account may not be opened or shared by multiple people or an automated software application
(e.g., an Internet bot).
You expressly understand and agree that your continued use of any Services after that
date shall be your automatic acceptance of this Agreement.

According to https://scan.coverity.com/policy

You will use the Software only in accordance with this Agreement and the applicable Registered Project Acknowledgement; and You will not use the Software to store or transmit any malicious code, interfere with or disrupt the integrity of the Software, or attempt to gain unauthorized access to the Software or its related systems or networks.

whatever that means :-)

[josepalafox]

I know the folks at Coverity working on their inventory if its helpful I can have them come to an openSSF meeting. LMK.

[evverx]

I'm not sure what the inventory is but it would be great if they could make reports really public by default when "Project summary and defects are viewable in read-only mode" is set to true. The systemd CI tends to send emails to contributors who it thinks introduce new defects and if they aren't particularly familiar with Coverity they usually end up clicking on the "add me to the project" button and waiting for the maintainers to approve their requests. At some point the documentation was updated:

Access to Coverity and oss-fuzz reports is limited. Please reach out to the maintainers if you need access.

but it's probably hard to find and I assume most "drive-by" contributors don't go out of their way to just be able to look at those reports. I think it's more important in general than access to their API (though I agree it would be nice if their licence could be reworded to make it safe to use scorecard).

[evverx]

I forgot to say that the previous comment is more or less applicable to CodeQL/Scorecard Action (or any other action utilizing the security tab) as well. It would be great if that dashboard could be made partly public. Though it should probably go to #1352.

[laurentsimon]

I'm on the same page, @evverx . I meant to ask @josepalafox if there is a chance the CodeQl results or the scanning dashboard could be made public in the future? This could be a simple opt-in for users to opt in. I see that lgtm.com already runs CodeQl for free, that the OSSF Omega project is also going to run CodeQl for open-source projects, and that anyone can run CodeQl on existing repos by forking them and running the workflow periodically.

So it seems like the data is already public in a sense. As GitHub already runs CodeQl, would it make sense to make the results public (opt-in) instead of duplicating this effort?

[evverx]

the OSSF Omega project is also going to run CodeQl for open-source projects

@laurentsimon out of curiosity was the Alpha-Omega project approved in the end? It's just that I'm not sure I agree with the Alpha part at least and it would be interesting to figure out where it's going.

[evverx]

Just to clarify, what I know about it is based on the public proposal I came across in December I think. At the time it was a draft and it could be that it might have changed and I don't know about it.

[laurentsimon]

it's going ahead. I think the official announcement will be made in March.

[evverx]

Got it. Thanks! Hopefully those one-off "long-term" engagements will be more helpful than they usually are.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[justaugustus]

@laurentsimon — we discussed in today's meeting and we're generally in disagreement on this addition.

I'm going to close it, especially given the age, but feel free to reopen the discussion.