tokens duplicates text if stream has chunks that end with escape character

[alec-hauck]

I ran into an issue where reading in json file of filepaths would sometimes duplicate text when running it through tokens. The following code when run with the attached sample text reproduces the issue. The second token is parsed as \pathpath\to\some\other\file.txt instead of \path\to\some\other\file.txt with chunkSize of 55

val chunkSize = 55
val stream: fs2.Stream[IO, Byte] = fs2.io.readInputStream(IO(getClass.getResourceAsStream("/Sample.txt")), chunkSize)
stream.map(_.toChar)
  .through(tokens[IO, Char])
  .map((token: Token) => {
    println(s"token: ${token}")
    token
  }).compile.toList.unsafeRunSync()

Sample.txt

I added some code to print out the chunks to see where they start and stop and found that when the chunks end with an escape character the text just before it gets duplicated on the next chunk.

val chunkSize = 55
val stream: fs2.Stream[IO, Byte] = fs2.io.readInputStream(IO(getClass.getResourceAsStream("/Sample.txt")), chunkSize)
stream.map(_.toChar).chunkN(chunkSize)
  .map((chunk: Chunk[Char]) => {
    println(s"chunk: ${chunk.toList.toString()}")
    chunk
  })
  .flatMap(c => fs2.Stream.chunk(c))
  .through(tokens[IO, Char])
  .map((token: Token) => {
    println(s"token: ${token}")
    token
  }).compile.toList.unsafeRunSync()

[ybasket]

Hey @alec-hauck,

thank you for reporting the issue and for the reproducer!
I think I found the issue and fixed in #516, at least it fixes your example. Let's see what @satabin has to say when reviewing, maybe there are even more edge cases like it.