[Misc] Unify tokenizer registration

[DarkLight1337]

Purpose

• Use tokenizer_mode instead of tokenizer_name as the registry ID, with temporary backward compatibility.
• In order to simplify the logic of get_tokenizer, also register HfTokenizer and MistralTokenizer to TokenizerRegistry.
• Moved get_tokenizer to vllm.tokenizers. The old location will be deprecated in the next PR.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request effectively unifies tokenizer registration by moving get_tokenizer to vllm.tokenizers and registering in-tree tokenizers like HfTokenizer and MistralTokenizer using a new decorator-based mechanism. This refactoring simplifies the tokenizer loading logic. I have identified two high-severity issues: one related to a breaking change that lacks a proper deprecation cycle, and another concerning a logic flaw in the new TokenizerRegistry.register method that could lead to silent failures. Addressing these will improve backward compatibility and the robustness of the new API.