[Chore] Use tokenizer.encode and tokenizer.decode directly

[DarkLight1337]

Purpose

Now that we have standardized the Tokenizer interface, we can replace encode_tokens and decode_tokens with tokenizer.encode and tokenizer.decode respectively.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request is a good refactoring effort to standardize the tokenizer interface by replacing encode_tokens and decode_tokens with direct calls to tokenizer.encode and tokenizer.decode. The changes are mostly correct and the deprecation of the old functions is appropriate. However, I've identified a critical bug in one of the test files where an incorrect number of arguments is passed to tokenizer.encode, which will cause the test to fail.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".