RFC: fusion should be importable by compiler sources, and even stdlib and koch

[timotheecour]

/cc @Araq

rationale

avoid code duplication and re-inventing the wheel

how it'd work

• build_all.sh builds csources
• the it builds koch
• then it calls koch boot
• koch boot clones dependencies in dist including fusion at a fixed hash baked in koch.nim (equivalent to a git submodule)
• import fusion/foo is made available for import, in fact it already is implemented in:

proc bundleFusion(latest: bool) =
  let commit = if latest: "HEAD" else: FusionStableCommit
  cloneDependency(distDir, "https://github.com/nim-lang/fusion.git", commit,
                  allowBundled = true)
  copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion")

• then nim c compiler/nim can use import fusion/foo
• likewise with stdlib modules; it's just a question of avoiding cyclic dependencies as usual except now there are 2 base dirs.

koch importing fusion

• that's also possible if fusion is cloned before koch.nim is built, ie via a git submodule or in build_all.sh; see for example add walkDirRecFilter recursive glob with follow filter (has full implementation) RFCs#261 where kochdocs could import fusion/globs

note

instead of copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion"), we should probably use the usual nimble logic which allows adding import dirs to nim's --path, so that nimble develop fusion works as intended (NIMBLE_DIR=customdir can always be used during bootstrap to avoid user's environment messing up with bootstrap)

[Araq]

Nah. Recursive dependencies between fusion and std mean there is no point in "fusion". And maybe indeed there isn't but then argue for that instead. :-)

[timotheecour]

can we at least agree on compiler/, tools/ and anything except koch and stdlib can import fusion?

Note that a recursive dependency between fusion and std doesn't imply cyclic module dependencies; it does add some package-level complexity but reduces likelihood of duplication.

argue for that instead

the way I see it, fusion is useful for following reasons:

• avoid a proliferation of small nimble packages (which tend to be abandoned by their author, become out of sync / incompatible) as often found in npm, eg: https://www.npmjs.com/package/is-negative; small modules are fine; small packages just adds many dependencies which people tend to avoid
• fusion modules work together (no incompatibility), maximize code reuse amongst them, minimizes redundancy
• nim devel tests against it and fusion tests against each release + nim devel (or at least should / will eventually)
• single docs (pending https://forum.nim-lang.org/t/6904) makes things more searchable
• finally, it's (IMO) intended to grow much larger than stdlib in terms of scope, as well as provide a testbed for new stdlib features

[alaviss]

instead of copyDir(distDir / "fusion" / "src" / "fusion", "lib" / "fusion"), we should probably use the usual nimble logic which allows adding import dirs to nim's --path, so that nimble develop fusion works as intended (NIMBLE_DIR=customdir can always be used during bootstrap to avoid user's environment messing up with bootstrap)

I decided against this and make use of copyDir simply because nimble does not support "system-installed" dependencies. Also it's widely incompatible with how Linux distributions package Nim, so unless we got those sorted out copying is the foolproof way.

[Araq]

can we at least agree on compiler/, tools/ and anything except koch and stdlib can import fusion?

tools are fine, I don't see why the compiler needs it just yet, maybe once fusion becomes more useful with support for compression etc.

[arnetheduck]

The absolutely simplest and most obvious solution here is that the compiler starts using a package manager - this solves pretty much everything and absolves the need for fusion and similar ideas that rest on contradictory premises.

• the compiler can lock specific versions of modules - this frees newer versions of packages in the standard library from having to be compatible with older versions of the compiler, purely for bootstrapping and it allows the compiler to progress at a separate pace from these packages - a packager_manager vendor command is used to copy the locked version of all those dependencies to get a "full, package-manager-free, offline, independent" version of the compiler at release time or whenever desired
• libraries that current sit in the standard library can be moved into separate packages, disconnecting them from the Nim release cycle and more easily allowing them to support multiple nim versions - this allows users of Nim to upgrade them individually more easily, for example when critical security fixes are made - it's often not possible to upgrade all of them together in reasonable time due to frequent regressions, if you consider the union of all changes across all packages.
• the same vendor command can be used to "assemble" a fat standard library for those that so desire without preventing them from using the packages individually
• the standard library can be stripped down to define the core of the language - magics, types etc - the rest can live in easily importable packages
• it's fine that some code in the compiler isn't always using the standard library but instead has its own implementation for critical features (like sets that could have been implemented libraries but weren't - for historical reasons) - that makes the two more independent in general which makes upgrades and maintenance easier because one doesn't have to consider the other when making changes.

Consider gcc and glibc - there's no expectation that one follows the other, and indeed, they each support a range of versions of the other. This flexibility allows them to progress at different paces - when a critical issue is identified in one, the other doesn't have to rush to make a release etc.

edit: It's important to point out that some of these packages can live in a single git repo as long as the PM supports it