Update on our foundational work

[squidfunk]

Dear community,

Many of you asked us about the status of our foundational work, including the new search. We want to give you an update on where we stand and what our plans are.

We're aware that more than a year has passed since our last blog post – we haven't been idle in the meantime. Technical and organizational challenges with MkDocs, our central upstream dependency, have made it impossible for us to push this project forward in the interest of our users and the wider community. This forced us to go back to the drawing board.

Still, we've been actively fixing bugs in Material for MkDocs as quickly as we can, but decided to postpone the release of new features – like the new search – with the project going into feature freeze. There are very good reasons for that, and we can finally talk about some of them, given the progress of our current efforts.

First and foremost, we and many of our users hit technical limitations in MkDocs that we couldn't sustainably work around anymore. We've had calls with many organizations bringing up the same unresolved problems in MkDocs over and over again, which is an upstream dependency for us, and thus software that we do not control. Many of those technical limitations are documented as issues and feature requests in MkDocs' issue tracker.

Historically, MkDocs has been maintained by a sequence of single maintainers, mostly in their spare time. In recent years, MkDocs seemed to be lacking a coherent vision, which led to the most pressing and biggest problems not being actively addressed. This has made it increasingly difficult for us to build and maintain a modern documentation toolchain on top of MkDocs.

In April 2024, the original author of MkDocs who left the project 10 years ago, announced that he would be stepping up as the lead maintainer again. However, aside from the 1.6.1 release in August 2024, there has been little to no progress on MkDocs. Issues, feature requests and PRs are piling up, effectively leaving MkDocs unmaintained. It's unclear what the maintainer has in mind for MkDocs, since the only thing publicly mentioned is work on a SaaS offering and a new theme. In regard to improving plugin support, the maintainer stated that he's working towards a version of MkDocs without plugins. Huge parts of MkDocs are implemented in plugins, including many of the features in Material for MkDocs, as well as key ecosystem projects like mkdocstrings.

This does not align with our vision of an open, extensible, and modern documentation toolchain that is self-hostable and fully Open Source. As you can imagine, we were shocked when we found out about these plans and their impact on our project, and quickly decided we need to find a way forward for Material for MkDocs and our users. We're fully committed to ensuring that you can continue to work on your projects with minimum friction, while benefitting from our foundational work that allows us to deliver the new features mentioned in our blog post, and much more.

We've been working on this new foundation for Material for MkDocs tirelessly for the last 16 months, pouring in thousands and thousands of hours. We're now, finally, very close to an initial release, including the new search, which we will announce before the end of the year. We will also be publishing our public roadmap.

If documentation plays a critical role in your organization, and you're worried how this might affect your business, please don't hesitate to reach out at [email protected] to arrange an enterprise feedback call.

You can subscribe to this thread to get notified when we announce. If you have any questions, please feel free to comment in this thread ☺️

Best always,
Martin, Alex & Kathi

[ofek]

Will all MkDocs plugins and python-markdown extensions still be compatible?

[alexvoss]

Compatibility and a smooth transition have been key concerns in our work. We will continue to support Python Markdown and its extensions and aim to ensure that most projects continue to build without modifications in configuration or Markdown.

We see the ecosystem as key to the attractiveness of Material for MkDocs and will be offering a significantly more powerful plugin system that will make it easier to write plugins. To overcome the technical limitations of MkDocs, there was no way around making some changes to its plugin API. However, our new API helps to reduce the amount of boilerpate and workarounds that are often necessary with MkDocs (as in several of our own plugins). Our work aims for a future where documentation toolchains are easy to adapt to a wide range of projects.

At the same time, we are aware of the existing investments and want to ensure a journey for everyone that is as smooth as possible. We will, of course, continue to maintain Material for MkDocs so people are not forced into changing their toolchain but we're sure that, gradually, everyone will want to transition in order to enjoy significantly faster builds, better search, and much more we'll share in the coming weeks and months.

[MateuszKubuszok]

This part:

This does not align with our vision of an open, extensible, and modern documentation toolchain that is self-hostable and fully Open Source. As you can imagine, we were shocked when we found out about these plans and their impact on our project, and quickly decided we need to find a way forward for Material for MkDocs and our users. We're fully committed to ensuring that you can continue to work on your projects with minimum friction, while benefitting from our foundational work that allows us to deliver the new features mentioned in our blog post, and much more.

sounds as if were planing a fork of upstream MkDocs? Is that the case or do you want to make sure that code keeps on working with MkDocs despite all these changes?

I am asking because I am using some plugins (mkdocs-macros) together with Material and I hope to keep using them together.

[squidfunk]

We've considered forking MkDocs, but that would mean saying goodbye to the entire plugin ecosystem. All MkDocs plugins depend on mkdocs as a dependency, which means they cannot just work with a fork. Thus, only the maintainers of MkDocs themselves could improve the existing plugin API, but there doesn't seem to be much interest in doing so, as outlined above.

Thus, we're at a crossroads: either accept the situation as it is, with MkDocs not moving forward, hindering our ability to innovate, or trying to improve on it, either by forking or rethinking things from the ground up. Since forking would not allow us to leverage the plugin ecosystem as it stands, we decided to go back to the drawing board and designed a far more modular plugin system. Additionally, any fundamental improvements to MkDocs' plugin API would also mean breaking changes with the need for the ecosystem to adapt. We'll also provide guidance on how to adapt existing MkDocs plugins easily to the new architecture.

You will, of course, still be able to use any MkDocs plugin with Material for MkDocs, as it's not going anywhere – we'll still keep maintaining it.

[kunickiaj]

While I love Material for MkDocs, I'm really disappointed with python-markdown and look forward to your upcoming announcement especially if it improves upon the underlying renderer!

[squidfunk]

Glad you love it! As mentioned before, we'll be compatible with Python Markdown, but we've also started exploring CommonMark as an alternative Markdown dialect to support, while keeping and expanding on the expressivity of Material for MkDocs. We're doing this, as it will drastically improve integration with the wider Markdown ecosystem.

[kunickiaj]

That is music to my ears and to everyone writing docs in our org!

[maxschulz-COL]

Thank you for this detailed message! And it goes without saying the myself and my team love Material for MkDocs.

I have a completely serious message I don't mean sarcastically: what does this mean concretely?

I was not able to identify from the message:

• will everything just improve for users - no user change required?
• will I need to change my codebase to keep using Material for MkDocs?
• will certain or all plugins not work anymore?

What is the takeaway for a professional user of Material for MkDocs?

Thanks again for this amazing tool!

[squidfunk]

Thanks for asking! As @alexvoss mentioned in #8461 (reply in thread), we'll provide full compatibility with Python Markdown, but the plugin system is a whole different story. Many of the problems of MkDocs that have not been adressed stem from the loose contracts its plugin system induces. This means that plugin interop is something that is quite impossible to achieve – I'm speaking from my own experience, having written 12 plugins shipped with Material for MkDocs. In fact, many MkDocs plugins have unsolved compatibility issues. We tried to raise the issues with the plugin system to the maintainer of MkDocs, but as I mentioned in my original message, there are no plans to address these issues – au contraire, removing the plugin system in favor of pure templating. Thus, there will be changes required, but we will provide a compatibility layer for our users to ease the transition.

As mentioned, we'll publish a detailed roadmap and more information on what this means for users of Material for MkDocs shortly. Please understand that all of this needs care, and we're only 3 people managing a user base of tens of thousands, so I kindly ask you to be patient with us.

Since you're working for @mckinsey, as mentioned in my original post, we also offer enterprise calls to our professional users.

[squidfunk]

Today, we're announcing Zensical – our new modern static site generator that reimagines static site generation from first principles. While Material for MkDocs is built on top of MkDocs, Zensical consolidates both projects into one coherent stack, covering static site generation, theming, and customization. Read the full announcement on our blog.