Deduplicate module file paths to prevent redundant scans.

[emcd]

Maintain and reference a set of visited files while expanding the module files from list of module directories, package names, and standalone files to prevent duplicate scans.

Type of Changes

| ✓ | 🐛 Bug fix |

Description

Closes #6242, #4053

[coveralls]

Pull Request Test Coverage Report for Build 3495785504

• 8 of 8 (100.0%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.0008%) to 95.413%

---

Totals
Change from base Build 3492123039:	0.0008%
Covered Lines:	17391
Relevant Lines:	18227

---

💛 - Coveralls

[Pierre-Sassoulas]

Thank you for contributing, much appreciated !

[Pierre-Sassoulas]

Could you check if it also fix #689, please ? Maybe we also need to sort the file to be stable between interpreters / operating system. Maybe a dict with an interpreter > 3.7 is enough.

[emcd]

Could you check if it also fix #689, please ? Maybe we also need to sort the file to be stable between interpreters / operating system. Maybe a dict with an interpreter > 3.7 is enough.

Do we want to sort the files? As a user of Pylint, I would prefer that it respects the order in which I specify arguments.
If we want reports to appear in a certain order, then it might be better to sort the reports rather the CLI arguments. (But I don't know how well that would interact with parallel jobs.)

Anyway, here's the output I get if I use the reproducer provided earlier this year:

$ pylint --score n --report n b.py a.py
************* Module b
b.py:6:0: C0115: Missing class docstring (missing-class-docstring)
b.py:6:0: C0103: Class name "B" doesn't conform to PascalCase naming style (invalid-name)
b.py:6:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module a
a.py:4:0: C0115: Missing class docstring (missing-class-docstring)
a.py:4:0: C0103: Class name "A" doesn't conform to PascalCase naming style (invalid-name)
a.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
a.py:5:4: C0104: Disallowed name "foo" (disallowed-name)
a.py:5:18: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
a.py:4:0: R0903: Too few public methods (1/2) (too-few-public-methods)

$ pylint --score n --report n a.py b.py
************* Module a
a.py:4:0: C0115: Missing class docstring (missing-class-docstring)
a.py:4:0: C0103: Class name "A" doesn't conform to PascalCase naming style (invalid-name)
a.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
a.py:5:4: C0104: Disallowed name "foo" (disallowed-name)
a.py:5:18: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
a.py:4:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module b
b.py:6:0: C0115: Missing class docstring (missing-class-docstring)
b.py:6:0: C0103: Class name "B" doesn't conform to PascalCase naming style (invalid-name)
b.py:6:0: R0903: Too few public methods (1/2) (too-few-public-methods)

So, the outputs are identical except for ordering. This looks right to me.

And, if I disable all of the checks that the reporter did not show, then I see:

$ pylint --score n --report n --disable=invalid-name,missing-function-docstring,missing-class-docstring,too-few-public-methods,disallowed-name b.py a.py
$ pylint --score n --report n --disable=invalid-name,missing-function-docstring,missing-class-docstring,too-few-public-methods,disallowed-name a.py b.py

I.e., identical output in spite of ordering. I do not know if this patch can take credit for the fix, but the issue does seem to be fixed one way or another.

[emcd]

Could you check if it also fix #689, please ? Maybe we also need to sort the file to be stable between interpreters / operating system. Maybe a dict with an interpreter > 3.7 is enough.

Do we want to sort the files? As a user of Pylint, I would prefer that it respects the order in which I specify arguments. If we want reports to appear in a certain order, then it might be better to sort the reports rather the CLI arguments. (But I don't know how well that would interact with parallel jobs.)

Anyway, here's the output I get if I use the reproducer provided earlier this year:

$ pylint --score n --report n b.py a.py
************* Module b
b.py:6:0: C0115: Missing class docstring (missing-class-docstring)
b.py:6:0: C0103: Class name "B" doesn't conform to PascalCase naming style (invalid-name)
b.py:6:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module a
a.py:4:0: C0115: Missing class docstring (missing-class-docstring)
a.py:4:0: C0103: Class name "A" doesn't conform to PascalCase naming style (invalid-name)
a.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
a.py:5:4: C0104: Disallowed name "foo" (disallowed-name)
a.py:5:18: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
a.py:4:0: R0903: Too few public methods (1/2) (too-few-public-methods)

$ pylint --score n --report n a.py b.py
************* Module a
a.py:4:0: C0115: Missing class docstring (missing-class-docstring)
a.py:4:0: C0103: Class name "A" doesn't conform to PascalCase naming style (invalid-name)
a.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
a.py:5:4: C0104: Disallowed name "foo" (disallowed-name)
a.py:5:18: C0103: Argument name "x" doesn't conform to snake_case naming style (invalid-name)
a.py:4:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module b
b.py:6:0: C0115: Missing class docstring (missing-class-docstring)
b.py:6:0: C0103: Class name "B" doesn't conform to PascalCase naming style (invalid-name)
b.py:6:0: R0903: Too few public methods (1/2) (too-few-public-methods)

So, the outputs are identical except for ordering. This looks right to me.

And, if I disable all of the checks that the reporter did not show, then I see:

$ pylint --score n --report n --disable=invalid-name,missing-function-docstring,missing-class-docstring,too-few-public-methods,disallowed-name b.py a.py
$ pylint --score n --report n --disable=invalid-name,missing-function-docstring,missing-class-docstring,too-few-public-methods,disallowed-name a.py b.py

I.e., identical output in spite of ordering. I do not know if this patch can take credit for the fix, but the issue does seem to be fixed one way or another.

I just checked against the other reproducer in #689 - there is still a problem with which check appears to be associated with which module, but there are no extra check message depending on order:

$ pylint --report n --score n mybase.py override.py
************* Module mymodule.mybase
mybase.py:5:0: C0115: Missing class docstring (missing-class-docstring)
************* Module mymodule.override
override.py:7:4: C0103: Attribute name "myInvalidName" doesn't conform to snake_case naming style (invalid-name)
mybase.py:5:0: R0903: Too few public methods (1/2) (too-few-public-methods)
override.py:5:0: C0116: Missing function or method docstring (missing-function-docstring)
override.py:6:4: C0104: Disallowed name "foo" (disallowed-name)

$ pylint --report n --score n override.py mybase.py
************* Module mymodule.override
override.py:5:0: C0116: Missing function or method docstring (missing-function-docstring)
override.py:6:4: C0104: Disallowed name "foo" (disallowed-name)
************* Module mymodule.mybase
mybase.py:5:0: C0115: Missing class docstring (missing-class-docstring)
override.py:7:4: C0103: Attribute name "myInvalidName" doesn't conform to snake_case naming style (invalid-name)
mybase.py:5:0: R0903: Too few public methods (1/2) (too-few-public-methods)

So, I think that #689 or some variant thereof must remain open.

[Pierre-Sassoulas]

LGTM, thank you. I have a suggestion to appease mypy, let me know what you think :)

[Pierre-Sassoulas]

Great first contribution, on a nasty bug that was open for far too long 👍 (Probably a pre-requisite to fix the multiprocessing too, another very important issue). Final nitpick and let's merge, maybe even in 2.15.6 :)

[github-actions]

🤖 According to the primer, this change has no effect on the checked open source code. 🤖🎉

This comment was generated for commit c67f204