Avoid merge conflicts in CHANGELOG.md

[jcrist]

With the current changelog policy (add an entry as part of the PR, all in one file) it's quite easy to run into merge conflicts as features are developed concurrently. This is not a problem unique to Prefect, many projects have run into this issue. There are three common solutions:

• Add a custom merge strategy for the CHANGELOG.md file (merge=union) to better handle these conflicts. This will usually work, but may run into issues if concurrently added features have very similar lines.
• Split the changelog into file-per-feature, and use a tool (towncrier is popular) to generate the final changelog from this before release.
• Add a bunch of filler entries (e.g. empty - list rows), and rely on developers adding entries in different spots to avoid merge conflicts.

[jcrist]

It looks like github doesn't currently support merge=union: isaacs/github#487.

[jcrist]

@jlowin, you self assigned this - are you still planning on taking this on? Otherwise I may hack on this at some point early next week.

[jlowin]

I sincerely apologize, I must have done that by mistake!

[jcrist]

So I got this working with towncrier, but before I finish up the PR I want to make sure we're happy with the functionality it provides. The workflow is:

• When making a change worth noting in the changelog, add a single file to a directory (configurable in name, we'll call it changes/ here).
• The filename you add is significant. The format should be {PR_NUMBER}.{KIND} where kind is a set of configurable kinds (e.g. bugfix, deprecation, feature, ...). For example, PR 1234 that adds a new feature would be 1234.feature. The file contains whatever content you'd put in the bullet point. We'd want to document the available file suffixes.
• The easiest way to make this work is that the compiled changelog only updates on release. This is done by running towncrier, which will use all the change files accrued in the changes/ directory to template in new content and add a new section to the CHANGELOG.md file. The files in the changes/ directory are then deleted. This is different than the existing workflow, where the changelog is always up to date, even with unreleased changes. If we wanted to have the changelog on the docs website (https://docs.prefect.io/api/latest/changelog.html) to be up to date with master, we could compile the changelog on docs build, but the static source in the repo would only be updated on release.

---

I think having individual contributors add a file to the changes/ directory as part of a PR is a fine mechanism, and helps prevent merge conflicts. I also think it's fine to not have the changelog update until a release (as noted above, we could keep the published version up to date, but not the CHANGELOG.md file).

I'm less sold on the formatting that towncrier requires. It makes the following harder to do:

• Have multiple bullets per PR (say a PR added a feature and deprecated an old one). It's possible, but you need to do it in two separate files.
• Keeping a list of contributors for each release is also hard. For dask we compile this from the git history since the last release (manual-ish, but this could be automated). We could also continue having contributors note themselves.

I'm not sure if the above dissatisfaction is enough to write our own script. Fundamentally compiling multiple changelog files into a single changelog on release isn't terribly complex, so creating a custom script wouldn't be a massive undertaking. But I want to get a yay/nay before I do anything else.

[jcrist]

If we were to make our own tool, I'd probably have the filename be the PR or issue number (for uniqueness, filename not significant). I'd also use yaml to structure the content. Something like:

# issue-1234.yaml
bugfix:
  - This is a changelog note, it's copied in verbatim as a bullet

deprecation:
  - can have multiple changelog notes per file, and multiple categories

[jcrist]

If we were to make our own tool, I'd probably have the filename be the PR or issue number (for uniqueness, filename not significant). I'd also use yaml to structure the content.

I went ahead and did this in #2529.