Candidates aren't displayed when vertico is triggered from emacs --eval, with kkp

[apodolsk]

Consider the following elisp:

;; (defmacro on-startup (&rest body)
;;   ;; Exploit the fact that post-command-hook also runs when first entering the command loop.
;;   `(add-hook 'post-command-hook
;;              (defun on-startup--hook ()
;;                (dbg (input-pending-p))
;;                (remove-hook 'post-command-hook 'on-startup--hook)
;;                ,@body)))

;; (defmacro on-startup (&rest body)
;;   `(run-with-idle-timer 0 nil
;;                         (lambda ()
;;                           (dbg (input-pending-p))
;;                           ,@body)))

(defmacro on-startup (&rest body)
  `(progn
     (while (input-pending-p)
       (message "input pending")
       (sleep-for 1))
     ,@body))

(defun with-initial-input (command initial-input)
  "Call interactive COMMAND with INITIAL-INPUT in the minibuffer."
  ;; (setq unread-command-events (listify-key-sequence initial-input))
  ;; (call-interactively command)
  (minibuffer-with-setup-hook
      (lambda ()
        (insert initial-input))
    (call-interactively command)))

When run via eval-expression, the expression (with-initial-input 'man "malloc") launches the man command, prepopulates input with "malloc", and displays completion candidates as expected. But when run via --eval, e.g. from the command emacs -nw --eval="(with-initial-input 'man \"malloc\")", it prepopulates input but doesn't display candidates until I type something else or run some other command.

With some printf debugging added to vertico.el, it seems that vertico--update is running as expected and seeing my spoofed input, but is bailing out and not running vertico--recompute because input-pending-p is apparently always true during --eval processing:

dbg: (minibuffer-contents-no-properties) = malloc
dbg: (input-pending-p) = t
dbg: (length lines) = 0

I included some alternate approaches and workarounds, including an on-startup function intended to be used to wait for input to stop pending e.g. via emacs -nw --eval="(on-startup (with-initial-input 'man \"malloc\"))". But those attempts all failed. The idle timer and post command hook run with input still pending, and the while loop hoping for async resolution of spuriously pending input never exits.

It's known that input-pending-p can return true to mean 'I dunno, maybe'. That does seem like it should be fixed in emacs itself, but there hasn't been a change since this came up and was filed in the very-related #226.

Do you know of any workarounds for this --eval case (maybe it would work to manually call vertico--recompute - I haven't yet tried)? If the point of the check is just to avoid wasted work during fast typing, could vertico maybe expose an option to disable it? Or maybe it's possible to get the same result with some non-false-positive check (looking at event queues like unread-command-events directly, maybe)?

This is kind of just a re-report of that linked issue, but it seems notable that this is a 100% repro that doesn't involve typing fast.

I'm hitting this because I want to turn that --eval command into a shell script to browse man pages in emacs, including showing pages relevant to a given query as completion candidates (since I sometimes get the page from the wrong man section, by default).

I'm running terminal emacs 30.2 in foot, in sway, in wayland.

[apodolsk]

Since the other issue was wayland only, I just tried my repro in urxvt+X11, and it happens there too.

[apodolsk]

vertico--recompute wasn't fit to yolo call, but discard-input does suffice as a workaround, clearing the spurious input-pending-p.

(defmacro on-startup (&rest body)
  `(progn
     ;; Work around https://github.com/minad/vertico/issues/635, where vertico doesn't update
     ;; candidate list due to spurious input-pending-p.
     (discard-input)
     ,@body))

[minad]

I think I've seen this problem before with programmatic execution of completion commands. Vertico relies on the post-command-hooks being triggered correctly and also pending events. If something is wrong there, Vertico will not refresh, but you can easily work around this by manually triggering the refresh.

[apodolsk]

This is apparently a bad interaction with kkp,el (https://github.com/benotn/kkp). My workaround above breaks it, and the issue is only present with kkp enabled. input-pending-p is apparently not spurious in this case - there are genuine kkp-related pending events (I can read-char them), it's just that processing them doesn't trigger post-command-hook.

In a tty-setup-hook, kkp.el communicates back and forth with the terminal emulator by sending and listening for special escape sequences. The 'listening' part is implemented by binding the expected response key sequence to a lambda in input-decode-map (https://github.com/benotn/kkp/blob/1a7b4f395aa4e1e04afc45fe2dbd6a045871803b/kkp.el#L534). It seems that input-decode-map function bindings are special in that they aren't run as commands, but are rather run as functions expected to return translated key sequences for further key translation. kkp's handler returns [], so further translation stops and no command or command hook is ultimately run.

You can repro this more directly with:

 (add-hook 'post-command-hook (lambda () (message "command hook: %s" this-command)))

;; Triggers `Wrong type argument: commandp`.
;; (define-key global-map (kbd "1")
;;             (lambda (&optional _prompt)
;;               (message "test")
;;               ))

;; Needs to be a key combo without any existing mappings in input-decode-map, hence a plain old
;; single char.
(define-key input-decode-map (kbd "1")
            (lambda (&optional _prompt)
              (message "input-decode-map handler")
              ;; Translate to zero keys, like kkp.
              []
              ))

If I type a and then a bunch of 1s, I see that the hook runs only for the self-insert-command:

a is undefined
command hook: undefined
input-decode-map handler [5 times]

This kkp.el logic is based on similar built-in emacs xterm handling, so it doesn't seem like it's doing something crazy.

Are you sure this doesn't qualify as a vertico bug? Apparently it's not safe to assume that input-pending-p implies that a command hook will eventually be run, as I think the redisplay logic does. Are 0-sec idle timers perhaps an option? Also, just skipping the input pending check on the initial redisplay of a session would at least fix my case, though it wouldn't fundamentally change things.

Also, how would one manually trigger the refresh? With some tracing, I can see that my problem is that vertico-recompute is getting skipped (the display code is running and finding my spoofed input, just skipping generating candidates for it), but my attempts to run it manually triggered errors and I don't think my inputs were wrong (it would take a bit of work to retry that experiment and show the exact error).

I do think that I could arrange to have my code wait until kkp is done with its negotiation, as it exposes its own hook.

[apodolsk]

Here's my latest workaround for posterity, using idle timers. This one doesn't break kkp.

I also needed to use the idle timer to work around a separate issue with zombie completing-read sessions irreversibly locking up the emacs daemon when running via emacsclient (like in https://emacs.stackexchange.com/questions/60256/if-an-emacsclient-has-an-open-minibuffer-when-its-closed-it-freezes-everything). But I assume that's just a generic emacs minibuffer problem, since I've been hitting variants of it for forever, including when using helm.

(defmacro run-after-startup (&rest body)
  "Defer BODY until problematic startup steps are finished. For use with --eval/-e."
  ;; If I don't defer body to idle at least once, then closing emacsclient via
  ;; save-buffers-kill-terminal (or just killing the terminal) while the minibuffer is still open
  ;; will lock up the emacs daemon.
  `(run-with-idle-timer
      0
      nil
      (defun-sym wait-for-input-clear ()
        ;; Defer some more to work around https://github.com/minad/vertico/issues/635, where vertico
        ;; doesn't update candidate list due to input-pending-p from kkp.
        ;;
        ;; It does eventually resolve between timer invocations, though it takes a few iterations.
        (if (input-pending-p)
            (run-with-idle-timer 0 nil 'wait-for-input-clear)
          ,@body))))