Screen.draw() silently drops text due to handling of SOH/STX

[moyix]

In bash apparently \[ and \] (common in shell prompts) get turned into SOH (0x01) and STX (0x02) respectively. However, I believe they are not being handled correctly by the parser in Stream, which results in sequences like \x02text being passed to Screen.draw(), which bails on the first control character it encounters (and therefore skips over text).

import pyte
def box(lines):
    w = len(lines[0])
    tb = f"+{'-'*w}+"
    return '\n'.join([tb]+[f'|{l}|' for l in lines]+[tb])
screen = pyte.Screen(10, 3)
stream = pyte.Stream(screen)
stream.feed('hello\r\n\x01dropped\x02dropped\r\n')
print(box(screen.display))
# Prints:
# +----------+
# |hello     |
# |          |
# |          |
# +----------+

The DebugStream gives:

["draw", ["hello"], {}]
["carriage_return", [], {}]
["linefeed", [], {}]
["draw", ["\u0001dropped\u0002dropped"], {}]
["carriage_return", [], {}]
["linefeed", [], {}]

The root cause seems to be the following regex to match "plain text":

pyte/pyte/streams.py

Lines 134 to 140 in 636b679

	#: A regular expression pattern matching everything what can be
	#: considered plain text.
	_special = set([ctrl.ESC, ctrl.CSI_C1, ctrl.NUL, ctrl.DEL, ctrl.OSC_C1])
	_special.update(basic)
	_text_pattern = re.compile(
	"[^" + "".join(map(re.escape, _special)) + "]+")
	del _special

Which is then used to find chunks of plain text that can be passed to draw():

pyte/pyte/streams.py

Lines 190 to 206 in 636b679

	draw = self.listener.draw
	match_text = self._text_pattern.match
	taking_plain_text = self._taking_plain_text
	length = len(data)
	offset = 0
	while offset < length:
	if taking_plain_text:
	match = match_text(data, offset)
	if match:
	start, offset = match.span()
	draw(data[start:offset])
	else:
	taking_plain_text = False
	else:
	taking_plain_text = send(data[offset:offset + 1])
	offset += 1

So a minimal solution (which works for me) would just be to add SOH and STX to the _special set so that they aren't treated as plain text. But maybe all of the control characters (0x00-0x1F) should be excluded as well?

Here is a quick patch. Happy to make a PR: master...moyix:pyte:moyix/fix_ctrl_chars

[deathaxe]

Text after control characters being dropped is caused by 1a9b914#diff-e059a3b34c367de4baecc0cbdce1847f5a268d08d65ae78128a4d856df7e298cR498

Before that change, draw() just ignored control characters. With this commit, text rendering is terminated, if a control character is detected (wcwidth() returns -1).

That's not a problem if draw() is called by parser for single characters, which turns the function into a NOOP:

pyte/pyte/streams.py

Line 381 in 61d0c0c

draw(char)

The following line is even redundant as draw() doesn't render CAN or SUB control characters:

pyte/pyte/streams.py

Line 338 in 61d0c0c

draw(char)

---

Stream._text_pattern seems primarily being used to terminate directly rendered chunks of plain text, if a known control or escape sequence is found, which needs to be sent to parser for it to generate events.

---

As draw() doesn't render and stops at control characters, those should indeed be catched by Stream. Those which do not trigger any event, should probably just be ignored, like NUL and DEL. It's even useless to call draw() for them.