gh-67230: add quoting rules to csv module

[smontanaro]

Add two quoting styles for csv dialects.
They will help to work with certain databases in particular.

• Issue: csv writer needs more quoting rules #67230

Automerge-Triggered-By: GH:merwok

[smontanaro]

I looked to try and find the real people behind the BPO accounts "samwyse" and "krypten" but couldn't. I have no idea how to verify at least krypten has signed a CLA.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[ned-deily]

I looked to try and find the real people behind the BPO accounts "samwyse" and "krypten" but couldn't. I have no idea how to verify at least krypten has signed a CLA.

@smontanaro, FWIW, according to the BPO user database, both of those users are registered as having signed the PSF CLA.

[MaxwellDupre]

You don't have empty quote '' in either of your tests:
self._write_test(['a',None,1], '"a",,1', quoting = csv.QUOTE_STRINGS)

self._write_test(['a',None,1], '"a",,"1"', quoting = csv.QUOTE_NOTNULL)

Hence I don't know if the code change produces the correct result.
I note that you can transpose the QUOTE_STRINGS and QUOTE_NOTNULL in these test and the affect is the same, and to be obvious you would only need one!
Also, since this is a little confusing (as to what is expected) could you add an example to the docs, please?
For each new quote so that users can see clearly whats expected and how they differ.

[samwyse]

@smontanaro, Is the CLA Signing waiting on me?

[AlexWaygood]

@smontanaro, Is the CLA Signing waiting on me?

Nope, all good, it just needed a new CI run to be retriggered, which I've just done by closing and reopening the PR :)

[smontanaro]

We need to fish or cut bait on this PR.

[msetina]

Why is merging blocked. These quoting types are essential for Microsoft warehousing CSV loading. Is there anyone form MS to push this to get better support for their loaders. CSV is broad format and these quoting types would fill some holes for special loaders without enough parameters to allow CSV prepared with Python.

[arhadthedev]

Why is merging blocked

Because only core developers can merge, and the one responsible for the csv module seems to go on vacation.

[msetina]

Is there any chance for this to land in 3.12?
I can elaborate why these additional quoting is needed:
As Python has a special value None It would be nice to have capability to control serialization of this to CSV. Currently it serializes into empty string quoted or not. But in some string columns/fields it is required on the parsing side that None is repesented differently than an empty string. If the reception side does not support parsing a special character set into None and it expects empty field without quoting even if field is a string and is quoted. This case is currently not supported in quoting possibilities, but is possible with new quoting rule. All of this is in realm of the RFC that should be supported by CSV module.
I mentioned Microsoft for this case as It is Synapse that has this kind of parser. Without additional postprocessing Synapse will not load CSV produced by Python CSV module. When working with Synapse team they stand by the RFC that their parser supports. I did express a whish to them that they support this change request, but...
I do not understand why it was not added in 3.11.

[smontanaro]

Sorry @msetina. All I can do is update the PR. I'll do what I can in that regard, but someone else will have to push to get it merged. @Mariatta do you have any suggestions?

[samwyse]

I realize that this is over a year old, but only noticed it today...

You don't have empty quote '' in either of your tests: self._write_test(['a',None,1], '"a",,1', quoting = csv.QUOTE_STRINGS)

self._write_test(['a',None,1], '"a",,"1"', quoting = csv.QUOTE_NOTNULL)

Change the test_csv.py to the following:

       self._write_test(['a','',None,1], '"a","",,1',
                        quoting = csv.QUOTE_STRINGS)
       self._write_test(['a','',None,1], '"a","",,"1"',
                        quoting = csv.QUOTE_NOTNULL)

I note that you can transpose the QUOTE_STRINGS and QUOTE_NOTNULL in these test and the affect is the same, and to be obvious you would only need one!

Not so! They differ in the treatment of numeric values. QUOTE_STRINGS would not quote numbers, so 1 is output as 1, while with QUOTE_NOTNULL it is output as "1".

Also, since this is a little confusing (as to what is expected) could you add an example to the docs, please? For each new quote so that users can see clearly whats expected and how they differ.

Let me see what I can do.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @merwok for commit ba74857 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[miss-islington]

@smontanaro: Status check is done, and it's a failure or timed out ❌.

[miss-islington]

@smontanaro: Status check is done, and it's a failure or timed out ❌.

[miss-islington]

@smontanaro: Status check is done, and it's a failure or timed out ❌.

[miss-islington]

Status check is done, and it's a success ✅.