reconsidering fenced divs

[brainchild0]

Problem

Fenced divs are an immensely useful selectable extension of the Markdown offered in Pandoc.

One limitation is that they have been defined as explicitly terminated. This property, though common in general markup languages and other computer languages, is opposed to the properties of all of the native block types of Markdown, which are terminated simply by suspending use of the syntax that marks them (as for lists and quotations), or are terminated simply by the beginning of a new block of similar type (as for sections). As such, fenced divs might be construed as a challenge to the design philosophy of the language they enhance.

As an example, in the following fragment, the "unquoted" block is indeed unquoted, and neither it nor the proceeding quoted block include or are included in any section that appears after them:

# One

> quoted

unquoted

## Primary
## Secondary

# Two

Indeed any sequence of plain, header, or quotation blocks may appear in any order, and while the order determines the structure of the document, each possible order corresponds one-to-one with a single valid result document.

Fenced divs break this property.

The following document is strictly wrong, even if processors are permissive enough to tolerate it silently:

# One

::: classname

Text

The processor might simply process the document as though the text ended in a terminator for the fenced div, as follows:

# One

::: classname

Text

:::

The value of such permissiveness becomes increasingly ambiguous a less trivial case, as follows:

# One

::: classname

Text

# Two

It is unclear how to resolve the relation between the fenced div and the trailing section, as the tree structure of the semantic document requires that if either node is a descendant of the other, then it cannot also be an ancestor.

Thus even a permissive processor, which might forgive the error silently, is unlikely to infer the intended meaning correctly in all cases. Even if the intention of the above might be considered to enclose only the plain "Text"" block in the fenced divs, such an inference is less obvious if, as follows, that block is dropped:

# One

::: classname

# Two

Then the fenced div might be empty, or might enclose the final section.

Notice the contrast between this property of fenced divs and the way sections are processed. The following two documents have different structures, but neither is intrinsically ambiguous:

# One

# Something else 

# Two

# One

## Something else 

# Two

In both cases, the final section terminates the previous, even though the level of the penultimate section differs in each case.

Fenced divs are currently construed as an item that may either descend from a section that begins earlier in the document, or enclose one that follows, and moreover, this determination relies entirely on the correct use of terminators. No item in standard Markdown has this property, and this construal is not only problematic, but avoidable.

Solution

Relationship to Sections

Much of the problem might be resolved by constraining fenced divs from enclosing sections. Whereas this restriction might seem harmful to some, all of the perceived harm is illusory, since for every practical case that sections enclosed in divs might appear to facilitate, a better solution is easily available by invoking alternative semantics.

Certainly, grouping sections of a document beyond the extent that is natural through a section hierarchy is conceptually a rarity.

Indeed, many sections need markings that are more specific than offered by their position in the section hierarchy, but such markings are already solved naturally in Pandoc Markdown through header attributes. If any set of sections has some attribute, then each each might simply be marked with that attribute. If processing them into an output format requires determining sequences of consecutive sections that share some attribute, then the processor may easily resolve such grouping without it being given explicitly in the source document.

Genuine grouping of sections is a rare document feature, though still deserving of support.

A clear example is parts of a book. Books that have parts divide main matter chapters into groups, while preserving the property of all books that chapters, not parts, are the top-level sections, and share a single numbering sequence across the entire book.

This grouping is naturally expressed simply by giving an attribute to the first chapter of each new part, and a further attribute, if needed, to the first chapter that is outside the final part. Another natural solution is naming the identifiers of the chapters that bound the parts of the book in the metadata of the document.

Indeed compared to the problems that arise in the eariler discussion, which would be even more serious in cases of fencing large sequences of many chapters in a book, the elimination of nested syntax in the document appears to represent a vast improvement

Suitable Syntax

Syntax Style A

If we tried to conceive of a design for fenced divs that more closely follows core Markdown, then we might consider one in which each line beginning with ::: simply resets the active set of classes to whichever are immediately specified on that line.

For example, if the text "bananas" is enclosed by the tropicalfencing, nested in the larger fruit fencing, then we could use syntax illustrated by the following:

# Food

::: fruit 

apples

::: fruit tropical

bananas

::: fruit

pears

# Clothes

etc.

While the above is not explicitly showing two levels of nested blocks, the class tags provide enough information to resolve that the text is enclosed in a region with the tropical class, nested within a larger region with the fruit class.

Syntax Style B

Or, if the above lacks an intuitive feel, we could consider a slightly different approach:

# Food

::: +fruit

apples

::: +tropical

bananas

::: -tropical

pears

# Clothes

etc.

Syntax Style C

In core Markdown, sections are the only items that bind to the trailing blocks even though those blocks are not each marked as belonging to the section.

For other types of blocks, such as lists and quotations, each line is separately marked as belonging to the block, otherwise, it assumed to be outside the block.

An analogous solution for fenced divs might merely augment the quotation syntax to include attributes :

# Food

> {.fruit} apples
>
>> {.tropical} bananas
>
> pears

# Clothes

etc.

Remarks

Of course the feasibility and popularity of replacing an entrenched feature seems doubtful, but such a course may be more likely than alternatives to support the long-term adoption and survival of the feature.

[mb21]

Maybe this is a better fit for pandoc-discuss?

[jgm]

Yes, better for pandoc-discuss.

We had a LONG discussion of fenced divs before the feature was added. (And there are other discussions on talk.commonmark.org.)

In all these discussions I advocated the idea of simply overloading the existing block quote syntax, but received heavy opposition. Fenced divs are a pain in some ways, but people really wanted them.

If we have fenced div openers, I think it's good to ask that they be explicitly closed (falling back to another behavior only as a kind of last-ditch fallback). I don't see that you get any advantage in readability by allowing them to be implicitly closed, and Markdown prioritizes readability after all.

[brainchild0]

@jgm: What are the references for the relevant earlier discussions?

I have only found one: https://groups.google.com/forum/#!topic/pandoc-discuss/_e_g7O_cEEo

[brainchild0]

I would like to review the earlier material relating to this discussion. I have found only one topic (see previous comment) in the Google discussion group, which does not match the description of being very long or expressing deep resistance to an approach that extends existing block-quotation syntax.

It would be helpful if @jgm or anyone else references a complete or at least comprehensive set of discussion that is affecting how the fenced divs topic currently is being considered.

(If you have difficulty finding a URL for some discussion in the Google group, then try to extract the topic identifier from the browser address bar and substitute it into the earlier-provided link.)

[tarleb]

It would be helpful if @jgm or anyone else references a complete or at least comprehensive set of discussion that is affecting how the fenced divs topic currently is being considered.

Note that, in this case, "anyone else", might as well be you.

[brainchild0]

@tarleb: I thought I made it clear that despite attempting to discover more I found only one discussion.

[tarleb]

Right, apologies for being overly brief. What I meant to imply is that the time required is to collect the information is probably roughly the same for everyone. Furthermore, I should have included in my reply the fact that searching the web for commonmark "div block syntax" and markdown "div block syntax" would be an excellent starting point to gather the required information.

[brainchild0]

@tarleb: The current discussion entails Pandoc features, and references a specific discussion or set of related discussions, described using the following language:

Yes, better for pandoc-discuss.

We had a LONG discussion of fenced divs before the feature was added.

I advocated the idea of simply overloading the existing block quote syntax, but received heavy opposition.

I have looked on pandoc-discuss for any discussion that I would characterize as a "long discussion on fenced divs" and that gave "heavy opposition" to "overloading the existing block syntax", and have found none.

All I want to know is whether someone can provide references to whatever discussions have affected the decisions over the Pandoc feature set adopted to date, beyond the one I have found.

Can you help?

[tarleb]

You seem to be acting under an incorrect premise. The mailing list is not the only place where such discussion might have happend. I recommend you try the search terms I suggested.

[brainchild0]

@tarleb: No, I don't believe that I have adopted the premise that the mailing list is the only place such a discussion might have happened. Rather I searched the mailing list for the discussion, and not having found it, I then asked where it did happen. Since you characterize the premise, if it were a premise, as incorrect, then you must know of a place other than the mailing list where the discussion might have happened. Would you please identify the place? Also, please understand that I am not currently having difficulty finding discussions about topics such as proposed CommonMark extensions for div blocks. The current situation is that I have not found, through your web queries, or any other methods, the location of the discussions that have been mentioned. Do you know the location of the discussions? If so, would you be kind enough to post the web links? Thank you.

[tarleb]

This is all the information I can offer. Since this doesn't seem to satisfy you, let me repeat an advice given to you previously: take it to pandoc-discuss.

[kysko]

The link to the mailing list syntax for generic spans and divs (2013-2014, 39 posts) you found looks like the right one.
It can be coupled with Issue #168 (2011-06+, 266 comments) which was implemented/resolved in fda0c01 (2017-10).

[brainchild0]

Thank you @kysko for the references. I had some doubts that the particular mailing-list topic was the one being referenced.

Having recently read, or reread, all of the discussion I have found, including both above, as well as several in the CommonMark forum, and noting that this topic has been closed, I now am seeking to understand how I might proceed in a way both that others find agreeable and that facilitates a fruitful conversation.

Primarily, I am trying to understand what will be accepted in terms of using the Github issue tracker versus the Google mailing list. The help page explains, "Suggestions for enhancements... should always be discussed on the mailing list before... on the github tracker." Although the language is clear, the practice may be murky.

I previously have opened enhancement discussions in the issue tracker, without objection. The Github interface offers numerous invaluable advantages over the Google web and email interfaces. On Github, it is relatively easily to write well-formatted and well-planned comments, and possible to edit comments and to reference other issues, wheres Google Groups has given me only headache, even when I compose externally and then copy into the internal editor.

Yet a deeper issue is the hope of finding interaction that seeks, through technical argument and discourse, solutions promising the best objective outcomes. As Github is principally a platform for developers, it is here where I hope for dialogue that is technically rich. Many discussions surrounding Pandoc, conversely, have directed at subjective preference or popularity. I am not dismissing this kind of conversation, as much as pointing to the value of other kinds.

As this topic plainly has not advanced as intended, further discussion would continue elsewhere, if at all. Under the circumstances, even though I still hold that the fencing approach leaves open serious vulnerabilities, I find most expedient putting it aside, at least for the time being, and directing efforts instead toward reopening the possibility of adding a side-marking syntax, as was discussed previously and summarized above in Style C, in a switchable extension that would have no interference with fencing. Since @jgm in the past has expressed enthusiasm about side marking, perhaps he would welcome such a discussion at this time. (I am only guessing. Perhaps he is tired of the whole thing.)

I would like to clarify whether I presently may submit such a suggestion to the issue tracker. If not, how might I understand, based on policy and practice, which submissions would be closed and which left open for discussion?

[mb21]

The idea is that technical- and implementation-details are discussed on GitHub, but big features (such as changes to the div syntax) should be discussed with pandoc's end-users and community, who don't necessarily watch all GitHub issues, that's why that should go on pandoc-discuss.

[brainchild0]

@mb21 I think I had that basic sense. I hesitated to express it in my last comment, because, referring to the help page beforehand, I discovered that the language was much more weighted away from using GitHub.

The distinction is fine, as you state it, but my feeling when opening this issue was that it was directed toward the technical side, expressing the ambiguities or vulnerabilities in the parser or processor design, as they relate to the resulting parse tree, which most end users are unlikely to consider seriously.

Ultimately the distinction opens a wide grey area between two extremes, where one must apply discretion, but I am struggling to understand how I might have predicted that the current topic would be judged as unsuitable for discussing in the issue tracker.

When I think about placing this article in the mailing list, I worry about being overwhelmed with personal opinions over which syntax is liked or disliked, which may be important to consider ultimately, but are beyond the scope of the topic.

[jgm]

We've made the decision long ago (after over a year of discussion). I'm not going to rethink it. There are lots of things we need to be focusing on right now, particularly supporting the new table features in readers and writers.

[brainchild0]

@jgm I understand the thoughts. Do you characterize the issue for which there is a lack of appetite to revisit as broadly encompassing the addition of a further side-marking syntax?

[jgm]

One could think about a new side-marking syntax, but it's not something I'd want to give energy to at the moment, as there are more pressing things to do.