RFC: Message extraction

[amannn]

Please use this thread to share feedback related to the RFC on message extraction.

This is my public TODO list for the feature: #2087 (but please keep the feedback here)

[amannn]

Question from @jurerotar:

A question I have is whether namespacing of messages will also be supported? I realize this is more of a SPA-related issue, but perhaps there are some use cases in Next.js apps where a developer is loading the localizations to the client and may want to split them up.

Answer:

Yep, namespaces are being considered, they are currently only briefly mentioned in regard to monorepos in the catalog generation section.

E.g.:

const t = useExtracted('namespace');
t('Hey there!');

… would generate e.g.:

{
  "namespace": {
    "5VpL9Z": "Hey there"
  }
}

I think the point is valid and this should be clarified more visibly (and perhaps not just be a "future exploration").

The ultimate goal for next-intl regarding splitting of messages for performance is #1, but as a stopgap solution some developers might find value in using some namespaces for this.

Does that answer the question?

[jurerotar]

Awesome, this looks really great!
Thank you very much for taking the time and clarifying this for me!

[amannn]

Quick poll: After reading the section on File formats & AI translation, which file format would you prefer as a default?

• 🚀 Portable object (.po)
• 🎉 Simple JSON (no contextual descriptions)
• 👀 Structured JSON (with vendor-specific format)

[amannn]

Yeah, there's a bit a tradeoff here I think. You could theoretically sort by file name, but if you move a translation from one file to another then there will also be an annoying diff where only the sorting changes. I'm not sure yet what the better option is …

[niieani]

btw. why no YAML option? (that's what we use)

[amannn]

Can you share an example snippet how this looks in detail? And is it based on a standard, or something you've defined for yourself?

[niieani]

Similar to your Structured JSON option, just in YAML. JSON has the issue of merge conflicts due to commas.
The output YAML is just a long array of strings, each defining the id, message and other options like the ones you would add in structured JSON. This is only to store the extracted strings. The consumable format that's downloaded by the app is either JSON files or JSON in JS files (extra trick for perf) + the default/fallback language embedded by the transpiler in the t() calls.

[amannn]

I see, got it! Yeah I think custom formatters will likely be necessary, just because people have different preferences here. Maybe support for some popular ones OOTB would be good though.

extra trick for perf

Super interesting, thanks for sharing!

[amannn]

Another poll: Do you prefer useExtracted or useInlined as the hook name?

import {useExtracted} from 'next-intl';

function InlineMessages() {
  const t = useExtracted();
  return <h1>{t('Look ma, no keys!')}</h1>;
}

import {useInlined} from 'next-intl';

function InlineMessages() {
  const t = useInlined();
  return <h1>{t('Look ma, no keys!')}</h1>;
}

Please vote with your preference:

• 🚀 useExtracted
• 🎉 useInlined

I'd also appreciate if you leave a comment here explaining why you prefer a particular option! (or why the alternative is bad)

[raphaelbadia]

I voted for useExtracted because it sounds good to me, I understand clearly what it means.
I feel like I would forget the name useInline, and I'd be like "What's the function called again" ?

[jurerotar]

This is a bit more on the hacky side, but how about adding the inline method to t directly?

import { useTranslations } from 'next-intl';

function InlineMessages() {
  const t = useTranslations();
  t.inline('Hello {name}', {name: 'Jane'});
}

T function signature is then either an invokable function or an object with this method. It doesn't address your third point though sadly :(

[amannn]

@raphaelbadia I can relate to that, yes!

@jurerotar Yep, theoretically also a possibility! However, next-intl currently also supports t.rich & t.markup and these will need to be supported for extracted messages too. I have a loose thought of getting rid of them in favor of a generic t function that would just return the right thing based on the arguments it receives, but it's still a bit further down the road—I wouldn't want it to block adoption of message extraction. So for the time being, message extraction should be compatible with those existing APIs.

[ceolinwill]

I don’t have a strong opinion, but useInline feels too vague since it could mean anything (not necessarily an inline translation). I’d probably prefer something like useInlineTranslation because it makes the purpose clearer.

[priyajeet]

Having the word Translation in the hook name will make the hook more self documented imo. Otherwise people might have to proxy hook to make it more readable. useInline and useExtracted don't really mean much imo unless you look at where its being imported from which may be collapsed away in github reviews.

[amannn]

Some thoughts on rich text formatting from @niieani:

As far as rich text, there's a way to do it while using string literals. I might have not mentioned that in my talk, but we had a solution, which was something like this:

const Bold = ({children}) => <b>{children}</b>
t(`This is <Bold>${userName}</Bold>.`, { components: { Bold } })

basically, we parse the string literal with the DOMParser and construct a React Array by wrapping any components with the components passed in as the component resolvers in the options passed to t function
this also works with nesting
and we statically analyze the strings during compilation to ensure that all the tags are correct and closed
this also means you can insert any component inside of any string, not just ones that wrap text (if that's something you needed) and is easy to understand by many translation tools.

I think it's an interesting approach, somewhere in the middle of using direct concatenation and using arguments in ICU strings. Kind of a hybrid solution in contrast to what the RFC currently discusses.

[amannn]

A first release is out in [email protected]! 🎉

Learn more:

• Blog post
• X post
• Bluesky post

[ceolinwill]

Thanks again for working on this. I wanted to share some initial feedback:

1. Source string in PO files

Right now, the msgid in PO files uses a random ID like Ju9oB+. This makes it harder to translate, since we have to check the source file to understand what each key refers to.

In other frameworks using Gettext (like Phoenix), the msgid stores the actual source string. For example:

msgid "Create an account"
msgstr ""

It would be great to have an option for this behavior, or at least include the source string as a comment above each msgid.

2. Deterministic compilation

Currently, every next build changes the order of items in the PO files, even when nothing in the code has changed. I tested this a few times and it happens consistently every time.

Ideally, the output should be deterministic and only update when translations are actually added, modified, or removed.

[amannn]

Many thanks for the feedback @ceolinwill!

This makes it harder to translate, since we have to check the source file to understand what each key refers to.

Where are you (planning to) edit your PO files? I found that e.g. Crowdin supports referencing source text from another locale automatically (the extractor adds X-Crowdin-SourceKey for this) and also local editors like Poedit support this.

While researching the feature, I found that some tools use source text as msgid with PO files while others use an ID. It seems to me like using an actual key here is just less redundant (we already have the source text in the source catalog) and it's kind of nice that it's shorter. The shorter nature is esp. handy in JSON files for consistent indentation (in comparison to e.g. "Hello there, how are you today?": "Hello there, how are you today?"), so being consistent across both JSON and PO might be another benefit. So it seems like it's mostly a tooling question (I'm also considering shipping a Cursor/VSCode extension at some point for local editing).

But that's just to share some background, it's absolutely valid to use source text as your ID if you prefer, I'm planning to support this through custom formatters (might be renamed to "adapters" or similar). I'm planning to open-source the PO parser I wrote for useExtracted, so with that, the only thing you'll need is a ~70 lines file like this: POFormatter.tsx and reference it in config like format: './po-formatter.js'.

I think it could be nice to publish this as a package as well that can be easily used and referenced by others. Since it would take slightly different tradeoffs than what I currently ship out of the box for useExtracted, I'd love to see this as a community package. Let me know if you'd like to publish it and I can help!

I'll keep this discussion updated once custom formatters are supported.

Currently, every next build changes the order of items in the PO files, even when nothing in the code has changed.

That's certainly a bug in case that happens. Are you able to reproduce this with https://github.com/amannn/next-intl-example-extracted? I haven't seen that so far. I'll check if I can find something suspicious, but if you could share a reproduction and ideally some info about your OS etc., that'd be really helpful!

[ceolinwill]

Where are you (planning to) edit your PO files? I found that e.g. Crowdin supports referencing source text from another locale automatically (the extractor adds X-Crowdin-SourceKey for this) and also local editors like Poedit support this.

Yeah, that approach works fine with Crowdin or Poedit. I usually work with code agents in my editor (like GitHub Copilot), where I point them to a file and ask them to add missing translations.

In those cases, having the source string helps because it reduces how much context the LLM has to process. Without it, I need to attach the source file too, and the model has to match IDs to strings, which increases token usage.

With source strings, LLMs can just read msgid "Create an account" and know what to translate, using fewer tokens.

It’s also easier to find or update a translation when browsing the repo on GitHub, since you can see the source text directly.

I'm planning to support this through custom formatters (might be renamed to "adapters" or similar)

Yeah, supporting custom formatters would definitely help with that.

That's certainly a bug in case that happens. Are you able to reproduce this with https://github.com/amannn/next-intl-example-extracted?

I wasn’t able to reproduce it there. I updated it to use po to check, but it worked fine.

Maybe it happens when there are lots of translations, or something in my setup triggers it. I won’t be able to look into it right now, but here’s the branch where I’ve seen the issue.

Getting it running might be tricky since it needs several env vars, but I’m using next-intl in the main app only. It also uses cacheComponents, so maybe the issue is related to that.

I’m also attaching a video showing the behavior:

next-intl-build-720.mov

[amannn]

Damn, that commit is 🔥! So cool to see all of that working, honestly! Did you do the conversion by hand / with an agent?

In those cases, having the source string helps because it reduces how much context the LLM has to process. Without it, I need to attach the source file too, and the model has to match IDs to strings, which increases token usage.

Got it, yes! My current thinking is that even looking at the catalog for a single locale is quite a bit of token waste. I think the broader point of services and tools like Crowdin and Poedit is that they allow very fine-grained usage of AI.

Something I'd like to experiment with is providing an MCP server along with tool calls like "get all missing translations", "add all missing translations" (compare e.g. the Next.js one). I think that could be a very efficient way for you to get this job done with minimal token usage. I'm still a bit new to the topic though, need some time to dig into this deeper …

I wasn’t able to reproduce it there

Let me investigate from the code perspective first, might be easier to diagnose if it's something small instead of spending a lot of time on a repro. I'll get back to you!

[ceolinwill]

Something else I just noticed:

Fallback for untranslated strings

Right now, untranslated strings render as empty. It would be great if they could fall back to the default locale.

For example, if a string isn’t yet translated to es, it could display the default version (eg en) instead of an empty value.

[amannn]

You can already do that in i18n/request.ts:

import {getRequestConfig} from 'next-intl/server';
 
export default getRequestConfig(async () => {
  const userLocale = 'es';
  const defaultLocale = 'en';

  const userMessages = (await import(`../../messages/${userLocale}.json`)).default;
  const defaultMessages = (await import(`../../messages/${defaultLocale}.json`)).default;
  const messages = {...defaultMessages, ...userMessages};

  return {
    messages,
    locale
  };
});

You can replace .json with .po depending on the formatter that you're using, but since this will import the messages as plain JS objects, the pattern is exactly the same.

In theory, we could display the source text in development as a fallback, but this would suddenly vanish once you build for production. So my current understanding is that it might be clearer if we let users do this themselves in either case, to have more flexibility and to avoid surprises.

Does that make sense?

[ceolinwill]

Yeah, that's a good idea. Thanks!

[bennajah]

Is the package tree-shakeable and does it bundle only the messages used by the current page — similar to how @inlang/paraglide-sveltekit works for SvelteKit?

[amannn]

Last time I checked, paraglide used per-message splitting. This is in contrast to next-intl, which uses per-locale splitting.

The former can help with tree shaking if you support a few locales, but the problem is that it will bundle all variants for all languages that you support. E.g. the Ethereum website localizes to 69 languages, therefore for projects of this size this is just not really an option to begin with.

With per-locale splitting, we need to solve the problem the other way round. Today, you can optimize manually if you like, but automatic tree-shaking is planned for the future.

The package itself is of course tree-shakeable (it's ESM-only) and will only bundle the features you really use.

Hope that helps!

[c0nsoleg]

Should it work with monorepo? Seems like message aren't picked up from packages at build or dev.

[amannn]

Monorepos are supported, yes. The docs are a bit sparse in regard to this though, I'm planning to include more.

See also: https://x.com/jamannnnnn/status/1987163883165860145

But did you encounter a bug? If so, can you share a reproduction?

[bennajah]

i dont have src folder and i want extract from root folder

so when use

srcPath: "./",

instead of

srcPath: ["./actions", "./app", "./components", "./lib", "./schemas"],

messages not extracted

[amannn]

Thanks, I'll look into this! I think it's likely happening due to a guard against traversing your node_modules missing, therefore the initial scan doesn't complete (or very late). I'll share an update here once this is fixed.

In the meantime, you can use an array of explicit folder names.

[priyajeet]

Will this work if t is passed into a plain old function?

[amannn]

Only statically analyzable calls like t('Hello!') are supported, anything dynamic like t(key) will print an error. I'll add some docs for this that covers how to write code in a statically analyzable way, covering common patterns.

Can you share your use case why you'd want to pass t into a function as a parameter?

[priyajeet]

Specifically for code organization. Say I want to translate based on an error code, I may have a function that takes the t function and code and has a gazillion switch case statements returning a different string based on code. And then I show this string to the user via a banner, say toast(). Likewise this function can be also used somewhere else. Now, one can make the function a hook and then useExtracted can be used in it, but then that is not re-usable in non-react parts and testing also becomes hook based as opposed to a simple plain old function based. The current t from useTranslations() seems to work when passed around (with some TS type annoyance), but it doesn't do any extraction so that was not an issue.

[bennajah]

Could you make locales accept a typed Locale? I'm getting a TypeScript error as shown in the screenshot below.
It’s not ideal to hardcode locales here since I’ve defined a reusable locales constant in i18n/config.ts