Port pure Sphinx site to Jupyter Book 1 (#846)

* Add advanced topics to _toc

* Working through optimization notebook

* Add script for putting examples into own notebook.

* Fix parsing of mathematical optimization notebook.

* Fix up parsing of some notebooks.

* Fix execution in some language notebooks.

* Edits to Scipy Sparse.

* Attempted edit to solve Sphinx build error.

* Working through pages.

* Maybe fix sparse solvers page.

* Another fix to solvers page.

* Remove image specifiers from guide

* Remove unused import

* Another couple of pages with wrong syntax.

* Remove ipython directive, fix table

* Exceptions rewrite without ipython directives.

* Remove ipython directive.

* Other uses if ipython

* ipython pygments syntax

* AUTHORS.md is a page; don't ignore.

* Rename substitution.

* Clean up basic_types

* Fix functions

* Move toctree into _toc

* Edit some warnings, for build fix

* Conmment out image directives.

* Reprocess image blocks for parsing.

* add sandbox file with early notes

* Fix build by removing stray LaTeX markup.

* Add editing instructions

* Work through advanced operations

* Fix up Numpy exercises

* Remove now-redundant gallery.

* Rename exercises to avoid clash with headings

* Rework a few more notebooks

* first adapted optimization graphs are in

* Docstrings for examples2nb

* Undo commenting of matplotlib example blocks

* Replace non-example image usage.

* Some replacements in .md files.

* Script to run regex over file

Modify by changing regex in file.

* Modify examples2nb to allow default examples.

* Tweak parsing, add writing of example notebooks.

* Literal for UnivariateSpline

* Rework script.

* add encoding check to examples script to sidestep error

* first pass at adaptive gradient descent adaptation

* first pass at all graphs on optimization page

* first pass at all graphs on optimization page

* Nearer final version of script

* Minor refactor

* Typo

* Fix up stats-interpolate exercises

* Working on the scikit-learn page.

* Set default notebook encoding.

* Remove prints

* Finish scikit-learn page first draft.

* Finish first pass edits of scikit-learn page.

* Recursive search for examples

and add ability to specify more than one directory to search for
examples.

* Add matplotlib examples notebook.

* Starting on Matplotlib tutorial.

* Working through Matplotlib foothills.

* update plotting code and add cell tags

* remove junk and update _scripts/

* Swap new optimization notebook for old.

* Finish up Matplotlib page.

* Remove examples page; now consumed.

* Fix build of debugging page

* Drop reference to sphinx-gallery in examples.

* Patchup of scikit-image page

There are many plots that need converting, but as we are rewriting this
page, let's leave those for now.

Look for `auto_examples` to find plots to convert.

* Work through statistics page

* Add stats examples notebook.

* Add Scipy examples

* Use ::: syntax for include.

* Remove "> " and some other edits.

* Run post_parser having stripped preceding "> "

* Remove stray "^>" markers.

* Working through Scipy page.

* Working through examples.

* Continuing Scipy page.

* Finish up Scipy page.

* Tweaks to improve build.

* Clean up some build failures

* Remove now-unused CSS

* Fixing some failures.

* Restore advanced_numpy page

Truncated by test edit.

* Move Matplotlib illustrations into own file.

* Replace ```{image} with ::: {image} in Numpy page.

* Update Numpy pages, fixing exercises.

* Add title to about page

To silence various warnings.

* Fix some references.

* Add data file for Numpy exercise

* Move note heading to block of note.

* Fix some scipy page references

* Use glue to pick up spectrogram example figures.

* Put Scipy FFT code back into examples.

* Try explicit title for about page.

* Remove reference to Numpy 1.13

It's too old now.

* Clean up IPython blocks

Change from python to ipython, restore prompts.

* Various reference fixes, URL fix.

* Fix some more reference errors.

* Clean up Python language section

* Clear up _toc

* Various fixes to build.

* Drop test workflow, install build deps for lint

* Pre-commit changes.

* Install build requirements for pages

* Remove unused _course.yml

* Add todo file.

* remove substitutions, fix captions and remove list tables from optimization page

* A couple of workflow fixes

Including nasty dependency resolution problem for pages.

* Todo style fixes.

* make intro more consistent with original, catch typo

* remove junk

* fix missing caption on optimization page

* Remove duplicate helper directory.

* Ruff suggestions

* Satisfy mypy

* Fix trailing whitespace.

* Add Linting make target.

* finish types page proofing, small changes

* Fix build directory

* proof control page, small changes

* proofing control page, more small edits

* proof functions page, small edits

* Adapt processing script to error processing

* Minor edits for intro page

Incidental to debugging processing script.

* Removing, fixing some quote markers

* implement most of Matthew's comments

* small edits to numpy array page

* Linter changes

* Fix tests to adapt to change in function API

* Neater way of getting first following node.

* Add test target.

* Adapt CONTRIBUTING to content.

* Move pre-commit into build_requirements.

* Reformat file with Ruff

* Ignore generated file.

* Refactor functions, add exercises[

and remove exercises from intro/scipy/solutions

* Rework control flow, add exercise.

* Note to look for exercises

* Remove last "> " remnants.

* Rework standard library page, clean solutions.

* Remove done todo item.

* Restore previous deletion of sidebar directives

* Remove some rst markup

* Working through exercises

Moving to sphinx-exercise

* Fix path to example data file.

* Exercises in optimization page.

* Exercises in optimization page.

* Complete removal of old exercise markers.

* Remove last ReST directives.

* Reworking skimage page.

* Finish up port of scikit-image page.

* Note completed task, add new task.

* fix tables

* remove junk image

* missing .

* Remove generated file from repo.

* Don't process .ipynb_checkpoints

* Label some generated files.

* Remove .ipynb_checkpoint file.

* make clean removes .ipynb_checkpoints directories

* Remove a :math: reference.

* Replace :ref: with {ref}

* Replace :func: with {func}

* Replace :mod: with {mod}

* Replace :class: with {class}

* Add intersphinx mappings.

* Fix scikit-image intersphinx mappings.

* Remove module reference that wasn't a module ref.

* Fix new reference label failure

Exposed by fixing :ref: -> {ref}

* Fix some download links.

* Fix a few class references.

* Remove extra space at beginning of line.

* -rf to avoid error deleting directories.

* Point to my copy for now.

* Remove use of <br> now I've found list-table

* Hide some inputs

* Full path for interact buttons.

* Refactor optimization page to list-tables and glue

The plots looked good, but the code was getting complicated and hard to
read on the main page.

* Remove now-uneeded header code in math-opt nbook.

* Fix repeated plot example; add plot code links.

* Fix Scipy module table.

* Change guide from ReST example to Markdown

* Add myst_nb to requirements

We need it for building pages with "glue" references.

* Use glue for camel graphic

* Note mathematical optimization done.

* Remove references to `rst` files.

* Remove last ".. " directive reference.

* Ignore joblib directories

* Update PRESENTING to be agnostic about build.

* We no longer use sphinx-gallery

* Remove conf.py

* Run compare_optimizers and commit rebuilt pkl

* Ruff reformat

* Fix a title.

* Fix link to plot.

* Rending -> rendering

* Initial port to Myst Markdown text format

* Generalize notebook processing for input formats.

* Add antialiased back to MPL quick ref table.

* Add prefix to interact URL

* Remove subdirectory spec for repository

This is not the subdirectory for the URL, but for the local files.

* Fix references to "md:myst" - should be "myst"

* Refactor processing to allowe remove-cell drop

* Lint

* Move regex around

* Add Myst jupyterlab / jupyterlite extension.

* Turn of processing of admonitions.

* Drop out glue directives.

* More complete message for glue refs.

* Refactor process_admonitions

* Clean joblib files

* Some tweaks to the TOC

* Add TOC to front page.

* Small capitalization and style fixes.

* Fix inclusion of metadata in included pages.

* Remove reference to PDF version

Does anyone use these?

* Add first tests.

* Explain testing, more on post-processing

* Allow Makefile clean commands to fail.

* Move Jupyterlab-myst to requirements.

* Satisfy linter.

* Note default True for proc_admonitions

* Remove typo from README.md

* Decapitalize some admonition directives

* Remove stub index pages from _toc, leave as redirects.

* Emphasise table captions

* Turn bold text into headings

* Switch URLs back to main site

* Maybe fix circle CI

* Try further fixes to circle-ci

* Refactor requirements

Previous requirements forcing older versions of jupyter-book.

* Revert "Switch URLs back to main site"

This reverts commit 6953285.

* Pin to Jupyter Book 1.

* Removed forced upgrade of jupyter-book

Now that would get us Jupyter Book 2.

---------

Co-authored-by: Peter Rush (Psychology) <[email protected]>

Author: matthew-brett

.circleci/config.yml

@@ -3,22 +3,11 @@ version: 2.1
 jobs:
   build:
     docker:
-      - image: cimg/python:3.12
+      - image: cimg/python:3.13
 
     steps:
       - checkout
 
-      - run:
-          name: Install TeX
-          command: |
-            sudo apt update
-            sudo apt install -y \
-              dvipng \
-              latexmk \
-              texlive-latex-extra \
-              texlive-fonts-extra \
-              texlive-extra-utils
-
       - restore_cache:
           keys:
             - pip-cache-v1
@@ -27,7 +16,7 @@ jobs:
           name: Install Python dependencies
           command: |
             pip install --upgrade --user pip
-            pip install --user -r requirements.txt
+            pip install --user -r build_requirements.txt
             pip list
 
       - save_cache:
@@ -40,12 +29,7 @@ jobs:
           command: |
             # NOTE: bad interaction w/ blas multithreading on circleci
             export OMP_NUM_THREADS=1
-            make pdf
-            make html  # FIX: check that failing examples produce failure
-            cp \
-              ScientificPythonLectures.pdf \
-              ScientificPythonLectures-simple.pdf \
-              build/html/_downloads/
+            make web  # FIX: check that failing examples produce failure
 
       - store_artifacts:
-          path: build/html
+          path: _build/html

.codespellrc

@@ -0,0 +1,3 @@
+[codespell]
+skip = .git,*.pdf,*.svg,*.csv
+ignore-words-list = trough,remainers,befores

.github/workflows/lint.yml

@@ -27,7 +27,7 @@ jobs:
       - name: Install packages
         run: |
           pip install --upgrade pip wheel setuptools
-          pip install -r requirements.txt
+          pip install -r build_requirements.txt
           pip list
 
       - name: Lint

.github/workflows/pages.yml

@@ -27,26 +27,21 @@ jobs:
       - name: Install Python dependencies
         run: |
           python -m pip install --upgrade pip wheel setuptools
-          python -m pip install -r requirements.txt
+          python -m pip install -r build_requirements.txt
 
-      - name: "Build PDF & HTML"
+      - name: "Build HTML"
         run: |
-          make pdf
           make html
-          mv \
-              ScientificPythonLectures.pdf \
-              ScientificPythonLectures-simple.pdf \
-              build/html/_downloads
-          echo -n 'lectures.scientific-python.org' > build/html/CNAME
-          touch build/html/.nojekyll
+          echo -n 'lectures.scientific-python.org' > _build/html/CNAME
+          touch _build/html/.nojekyll
 
       - name: Deploy
         uses: peaceiris/actions-gh-pages@v4
         with:
           deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
-          external_repository: scipy-lectures/lectures.scientific-python.org
+          # external_repository: scipy-lectures/lectures.scientific-python.org
           publish_branch: gh-pages
-          publish_dir: ./build/html
+          publish_dir: ./_build/html
           force_orphan: true
           user_name: "github-actions[bot]"
           user_email: "github-actions[bot]@users.noreply.github.com"

.github/workflows/test.yml

@@ -28,9 +28,13 @@ jobs:
 
       - name: Install packages
         run: |
-          python -m pip install --upgrade pip wheel setuptools
-          python -m pip install -r requirements.txt
+          python -m pip install -r test_requirements.txt
           python -m pip list
 
-      - name: Test lectures
-        run: make test
+      - name: Test
+        run: |
+          # Avoid deprecation error.
+          export JUPYTER_PLATFORM_DIRS=1
+          jupyter --paths
+
+          make test

.gitignore

@@ -45,3 +45,17 @@ airfares.txt
 wages.txt
 ScientificPythonLectures-simple.pdf
 ScientificPythonLectures.pdf
+.ipynb_checkpoints/
+__pycache__/
+.ok_storage
+*.ipynb
+*.orig
+node_modules/
+.jupyterlite.doit.db
+advanced/advanced_numpy/test.png
+packages/scikit-image/cat.png
+advanced/advanced_numpy/test_recolored.png
+advanced/advanced_numpy/test_red.png
+intro/language/junk.txt
+intro/language/test.pkl
+joblib/

.pre-commit-config.yaml

@@ -45,6 +45,8 @@ repos:
         additional_dependencies:
           - types-aiofiles
           - types-requests
+          - types-docutils
+          - types-PyYAML
           - pandas-stubs
           - types-pillow
           - matplotlib
@@ -56,6 +58,8 @@ repos:
             | .*/setup.*\.py$
             | .*/demo.py$
             | .*/auto_examples/
+            | _scripts/examples2nb.py$
+            | _scripts/post_parser.py$
             | advanced/mathematical_optimization/examples/plot_gradient_descent\.py$
             | advanced/mathematical_optimization/examples/helper/compare_optimizers\.py$
             | advanced/advanced_numpy/examples/view-colors\.py$