html5gum does not detect lone surrogates

[untitaker]

lone surrogates are invalid utf8, so html5gum 0.3.0, which takes &str/String, is not able to handle those.

after merging #25, html5gum will be able to read arbitrary bytes. at this point the expectation might be that lone surrogates produce error tokens, but they do not.

note: lone surrogates have no impact on parsing behavior. only some error tokens are missing from token stream.

[Ygg01]

Do you need to worry about lone surrogates? In order to use Vec<u8> you would need to convert it to an encoding via std::str::from_utf8

Only possible issue is if the stream is some non-UTF8 format.

[untitaker]

html5gum assumes that the incoming bytestream is mostly valid UTF-8. The spec says that lone surrogate should create an error, and currently they don't. I don't need to worry beyond that.