Merge tag 'v1.44.0rc1' into develop

Synapse 1.44.0rc1 (2021-09-29)
==============================

Features
--------

- Only allow the
[MSC2716](matrix-org/matrix-spec-proposals#2716)
`/batch_send?chunk_id=xxx` endpoint to connect to an already existing
insertion event.
([\#10776](#10776))
- Improve oEmbed URL previews by processing the author name, photo, and
video information.
([\#10814](#10814),
[\#10819](#10819))
- Speed up responding with large JSON objects to requests.
([\#10868](#10868),
[\#10905](#10905))
- Add a `user_may_create_room_with_invites` spam checker callback to
allow modules to allow or deny a room creation request based on the
invites and/or 3PID invites it includes.
([\#10898](#10898))

Bugfixes
--------

- Fix a long-standing bug that caused an `AssertionError` when purging
history in certain rooms. Contributed by @Kokokokoka.
([\#10690](#10690))
- Fix a long-standing bug which caused deactivated users that were later
reactivated to be missing from the user directory.
([\#10782](#10782))
- Fix a long-standing bug that caused unbanning a user by sending a
membership event to fail. Contributed by @aaronraimist.
([\#10807](#10807))
- Fix a long-standing bug where logging contexts would go missing when
federation requests time out.
([\#10810](#10810))
- Fix a long-standing bug causing an error in the deprecated
`/initialSync` endpoint when using the undocumented `from` and `to`
parameters.
([\#10827](#10827))
- Fix a bug causing the `remove_stale_pushers` background job to
repeatedly fail and log errors. This bug affected Synapse servers that
had been upgraded from version 1.28 or older and are using SQLite.
([\#10843](#10843))
- Fix a long-standing bug in Unicode support of the room search admin
API breaking search for rooms with non-ASCII characters.
([\#10859](#10859))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership
events which we sent to remote servers to be incorrectly stored in the
local database.
([\#10873](#10873))
- Fix invalidating one-time key count cache after claiming keys. The bug
was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
([\#10875](#10875))
- Fix a long-standing bug causing application service users to be
subject to MAU blocking if the MAU limit had been reached, even if
configured not to be blocked.
([\#10881](#10881))
- Fix a long-standing bug which could cause events pulled over
federation to be incorrectly rejected.
([\#10907](#10907))
- Fix a long-standing bug causing URL cache files to be stored in
storage providers. Server admins may safely delete the `url_cache/` and
`url_cache_thumbnails/` directories from any configured storage
providers to reclaim space.
([\#10911](#10911))
- Fix a long-standing bug leading to race conditions when creating media
store and config directories.
([\#10913](#10913))

Improved Documentation
----------------------

- Fix some crashes in the Module API example code, by adding JSON
encoding/decoding.
([\#10845](#10845))
- Add developer documentation about experimental configuration flags.
([\#10865](#10865))
- Properly remove deleted files from GitHub pages when generating the
documentation.
([\#10869](#10869))

Internal Changes
----------------

- Fix GitHub Actions config so we can run sytest on synapse from
parallel branches.
([\#10659](#10659))
- Split out
[MSC2716](matrix-org/matrix-spec-proposals#2716) meta
events to their own fields in the `/batch_send` response.
([\#10777](#10777))
- Add missing type hints to REST servlets.
([\#10785](#10785),
[\#10817](#10817))
- Simplify the internal logic which maintains the user directory
database tables.
([\#10796](#10796))
- Use direct references to config flags.
([\#10812](#10812),
[\#10885](#10885),
[\#10893](#10893),
[\#10897](#10897))
- Specify the type of token in generic "Invalid token" error messages.
([\#10815](#10815))
- Make `StateFilter` frozen so it is hashable.
([\#10816](#10816))
- Fix a long-standing bug where an `m.room.message` event containing a
null byte would cause an internal server error.
([\#10820](#10820))
- Add type hints to the state database.
([\#10823](#10823))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to
hopefully improve `/sync` performance when you
haven't synced recently.
([\#10826](#10826))
- Track cache eviction rates more finely in Prometheus's monitoring.
([\#10829](#10829))
- Add missing type hints to `synapse.handlers`.
([\#10831](#10831),
[\#10856](#10856))
- Extend the Module API to let plug-ins check whether an ID is local and
to access IP + User Agent data.
([\#10833](#10833))
- Factor out PNG image data to a constant to be used in several tests.
([\#10834](#10834))
- Add a test to ensure state events sent by modules get persisted
correctly.
([\#10835](#10835))
- Rename [MSC2716](matrix-org/matrix-spec-proposals#2716)
fields and event types from `chunk` to `batch` to match the
`/batch_send` endpoint.
([\#10838](#10838))
- Rename [MSC2716](matrix-org/matrix-spec-proposals#2716)
`/batch_send` query parameter from `?prev_event` to more obvious usage
with `?prev_event_id`.
([\#10839](#10839))
- Add type hints to `synapse.http.site`.
([\#10867](#10867))
- Include outlier status when we log V2 or V3 events.
([\#10879](#10879))
- Break down Grafana's cache expiry time series based on reason for
eviction, c.f.
[\#10829](#10829).
([\#10880](#10880))
- Clean up some of the federation event authentication code for clarity.
([\#10883](#10883),
[\#10884](#10884),
[\#10896](#10896),
[\#10901](#10901))
- Allow the `.` and `~` characters when creating registration tokens as
per the change to
[MSC3231](matrix-org/matrix-spec-proposals#3231).
([\#10887](#10887))
- Clean up some unnecessary parentheses in places around the codebase.
([\#10889](#10889))
- Improve type hinting in the user directory code.
([\#10891](#10891))
- Update development testing script `test_postgresql.sh` to use a
supported Python version and make re-runs quicker.
([\#10906](#10906))
- Document and summarize changes in schema version `61` – `64`.
([\#10917](#10917))
- Update release script to sign the newly created git tags.
([\#10925](#10925))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build
their docs.
([\#10931](#10931))

F124520CEEE062448FE1C8442D2EFA2F32FBE047
<[email protected]>" [ultimate]

Author: reivilibre

CHANGES.md

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.44.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 28 Sep 2021 13:41:28 +0100
+
 matrix-synapse-py3 (1.43.0) stable; urgency=medium
 
   * New synapse release 1.43.0.

changelog.d/10659.misc

@@ -47,7 +47,7 @@
 except ImportError:
     pass
 
-__version__ = "1.43.0"
+__version__ = "1.44.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

changelog.d/10690.bugfix

@@ -561,9 +561,17 @@ def __init__(
         self._iterator = iterator
         self._paused = False
 
-        # Register the producer and start producing data.
-        self._request.registerProducer(self, True)
-        self.resumeProducing()
+        try:
+            self._request.registerProducer(self, True)
+        except RuntimeError as e:
+            logger.info("Connection disconnected before response was written: %r", e)
+
+            # We drop our references to data we'll not use.
+            self._request = None
+            self._iterator = iter(())
+        else:
+            # Start producing if `registerProducer` was successful
+            self.resumeProducing()
 
     def _send_data(self, data: List[bytes]) -> None:
         """