Merge pull request #225 from ad-hardy/feature-custom-units-and-entities

custom units and entities

Author: nielstron

.travis.yml

@@ -51,7 +51,7 @@ script:
   - coverage run -a --source=quantulum3 setup.py test -s quantulum3.tests.test_classifier.ClassifierTest
   - coverage run -a --source=quantulum3 setup.py test -s quantulum3.tests.test_scripts.TrainScriptTest
   - coverage run -a --source=quantulum3 setup.py test -s quantulum3.tests.test_load.TestCached
-
+  - coverage run -a --source=quantulum3 setup.py test -s quantulum3.tests.test_load.TestLoaders
 after_success:
   - coverage report
   - coveralls

README.md

@@ -1,11 +1,11 @@
-quantulum3
-==========
+# quantulum3
+
  [![Travis master build state](https://app.travis-ci.com/nielstron/quantulum3.svg?branch=master "Travis master build state")](https://app.travis-ci.com/nielstron/quantulum3)
  [![Coverage Status](https://coveralls.io/repos/github/nielstron/quantulum3/badge.svg?branch=master)](https://coveralls.io/github/nielstron/quantulum3?branch=master)
  [![PyPI version](https://badge.fury.io/py/quantulum3.svg)](https://pypi.org/project/quantulum3/)
  ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/quantulum3.svg)
  [![PyPI - Status](https://img.shields.io/pypi/status/quantulum3.svg)](https://pypi.org/project/quantulum3/)
- 
+
 Python library for information extraction of quantities, measurements
 and their units from unstructured text. It is able to disambiguate between similar
 looking units based on their *k-nearest neighbours* in their [GloVe](https://nlp.stanford.edu/projects/glove/) vector representation
@@ -18,23 +18,23 @@ Lagi](https://github.com/marcolagi/quantulum).
 The compatibility with the newest version of sklearn is based on
 the fork of [sohrabtowfighi](https://github.com/sohrabtowfighi/quantulum).
 
-Installation
-------------
+## User Guide
+
+### Installation
 
 ```bash
-$ pip install quantulum3
+pip install quantulum3
 ```
 
 To install dependencies for using or training the disambiguation classifier, use
 
 ```bash
-$ pip install quantulum3[classifier]
+pip install quantulum3[classifier]
 ```
 
 The disambiguation classifier is used when the parser find two or more units that are a match for the text.
 
-Usage
------
+### Usage
 
 ```pycon
 >>> from quantulum3 import parser
@@ -79,8 +79,7 @@ this library can also be used for simple number extraction.
 [Quantity(2, 'dimensionless')]
 ```
 
-Units and entities
-------------------
+### Units and entities
 
 All units (e.g. *litre*) and the entities they are associated to (e.g.
 *volume*) are reconciled against WikiPedia:
@@ -117,8 +116,7 @@ dimensionality:
 Unit(name="kilometre per second", entity=Entity("speed"), uri=None)
 ```
 
-Disambiguation
---------------
+### Disambiguation
 
 If the parser detects an ambiguity, a classifier based on the WikiPedia
 pages of the ambiguous units or entities tries to guess the right one:
@@ -147,27 +145,47 @@ In addition to that, the classifier is trained on the most similar words to
 all of the units surfaces, according to their distance in [GloVe](https://nlp.stanford.edu/projects/glove/)
 vector representation.
 
-Training the classifier
------------------------
-
-If you want to train the classifier yourself, in addition to the packages above, you'll also need
-the packages `stemming` and `wikipedia`.
+## Spoken version
 
-You can get the classifier dependencies by running
+Quantulum classes include methods to convert them to a speakable unit.
 
-```bash
-$ pip install quantulum3[classifier]
+```pycon
+>>> parser.parse("Gimme 10e9 GW now!")[0].to_spoken()
+ten billion gigawatts
+>>> parser.inline_parse_and_expand("Gimme $1e10 now and also 1 TW and 0.5 J!")
+Gimme ten billion dollars now and also one terawatt and zero point five joules!
 ```
 
-You could also [download requirements_classifier.txt](https://raw.githubusercontent.com/nielstron/quantulum3/dev/requirements_classifier.txt)
-and run
+### Manipulation
 
-```bash
-$ pip install -r requirements_classifier.txt
-```
+While quantities cannot be manipulated within this library, there are
+many great options out there:
+
+- [pint](https://pint.readthedocs.org/en/latest/)
+- [natu](http://kdavies4.github.io/natu/)
+- [quantities](http://python-quantities.readthedocs.org/en/latest/)
+
+## Extension
+
+### Training the classifier
+
+If you want to train the classifier yourself, you will need the dependencies for the classifier (see installation).
 
 Use `quantulum3-training` on the command line, the script `quantulum3/scripts/train.py` or the method `train_classifier` in `quantulum3.classifier` to train the classifier.
 
+``` bash
+quantulum3-training --lang <language> --data <path/to/training/file.json> --output <path/to/output/file.joblib>
+```
+
+You can pass multiple training files in to the training command. The output is in joblib format.
+
+To use your custom model, pass the path to the trained model file to the
+parser:
+
+```pyton
+parser = Parser.parse(<text>, classifier_path="path/to/model.joblib")
+```
+
 Example training files can be found in `quantulum3/_lang/<language>/train`.
 
 If you want to create a new or different `similars.json`, install `pymagnitude`.
@@ -183,62 +201,62 @@ converted to a `.magnitude` file on-the-run. Check out
 [pre-formatted Magnitude formatted word-embeddings](https://github.com/plasticityai/magnitude#pre-converted-magnitude-formats-of-popular-embeddings-models)
 and [Magnitude](https://github.com/plasticityai/magnitude) for more information.
 
+### Additional units
 
-To use your custom model, pass the path to the trained model file to the
-parser:
+It is possible to add additional entities and units to be parsed by quantulum. These will be added to the default units and entities. See below code for an example invocation:
 
-```pyton
-parser = Parser.parse(classifier_path="path/to/model")
+```pycon
+>>> from quantulum3.load import add_custom_unit, remove_custom_unit
+>>> add_custom_unit(name="schlurp", surfaces=["slp"], entity="dimensionless")
+>>> parser.parse("This extremely sharp tool is precise up to 0.5 slp")
+[Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]
 ```
 
+The keyword arguments to the function `add_custom_unit` are directly translated
+to the properties of the unit to be created.
 
-Manipulation
-------------
-
-While quantities cannot be manipulated within this library, there are
-many great options out there:
-
--   [pint](https://pint.readthedocs.org/en/latest/)
--   [natu](http://kdavies4.github.io/natu/)
--   [quantities](http://python-quantities.readthedocs.org/en/latest/)
+### Custom Units and Entities
 
-Spoken version
---------------
+It is possible to load a completely custom set of units and entities. This can be done by passing a list of file paths to the load_custom_units and load_custom_entities functions. Loading custom untis and entities will replace the default units and entities that are normally loaded.
 
-Quantulum classes include methods to convert them to a speakable unit.
+The recomended way to load quantities is via a context manager:
 
 ```pycon
->>> parser.parse("Gimme 10e9 GW now!")[0].to_spoken()
-ten billion gigawatts
->>> parser.inline_parse_and_expand("Gimme $1e10 now and also 1 TW and 0.5 J!")
-Gimme ten billion dollars now and also one terawatt and zero point five joules!
-```
+>>> from quantulum3 import load, parser
+>>> with load.CustomQuantities(["path/to/units.json"], ["path/to/entities.json"]):
+>>>     parser.parse("This extremely sharp tool is precise up to 0.5 slp")
 
-Extension
----------
+[Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]
 
-#### Custom units
+>>> # default units and entities are loaded again
+```
 
-It is possible to add custom entities to be parsed by quantulum.
-See below code for an example invocation.
+But it is also possible to load custom units and entities manually:
 
 ```pycon
->>> from quantulum3.load import add_custom_unit, remove_custom_unit
->>> add_custom_unit(name="schlurp", surfaces=["slp"], entity="dimensionless")
+>>> from quantulum3 import load, parser
+
+>>> load.load_custom_units(["path/to/units.json"])
+>>> load.load_custom_entities(["path/to/entities.json"])
 >>> parser.parse("This extremely sharp tool is precise up to 0.5 slp")
+
 [Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]
+
+>>> # remove custom units and entities and load default units and entities
+>>> load.reset_quantities()
 ```
 
-The keyword arguments to the function `add_custom_unit` are directly translated
-to the properties of the unit to be created.
+See the Developer Guide below for more information about the format of units and entities files.
+
+## Developer Guide
 
-#### Extending the set of default units
+### Adding Units and Entities
 
 See *units.json* for the complete list of units and *entities.json* for
 the complete list of entities. The criteria for adding units have been:
 
--   the unit has (or is redirected to) a WikiPedia page
--   the unit is in common use (e.g. not the [premetric Swedish units of
+- the unit has (or is redirected to) a WikiPedia page
+- the unit is in common use (e.g. not the [premetric Swedish units of
     measurement](https://en.wikipedia.org/wiki/Swedish_units_of_measurement#Length)).
 
 It\'s easy to extend these two files to the units/entities of interest.
@@ -251,9 +269,9 @@ Here is an example of an entry in *entities.json*:
 }
 ```
 
--   The *name* of an entity is its key. Names are required to be unique.
--   *URI* is the name of the wikipedia page of the entity. (i.e. `https://en.wikipedia.org/wiki/Speed` => `Speed`)
--   *dimensions* is the dimensionality, a list of dictionaries each
+- The *name* of an entity is its key. Names are required to be unique.
+- *URI* is the name of the wikipedia page of the entity. (i.e. `https://en.wikipedia.org/wiki/Speed` => `Speed`)
+- *dimensions* is the dimensionality, a list of dictionaries each
     having a *base* (the name of another entity) and a *power* (an
     integer, can be negative).
 
@@ -277,24 +295,24 @@ Here is an example of an entry in *units.json*:
 }
 ```
 
--   The *name* of a unit is its key. Names are required to be unique.
--   *URI* follows the same scheme as in the *entities.json*
--   *surfaces* is a list of strings that refer to that unit. The library
+- The *name* of a unit is its key. Names are required to be unique.
+- *URI* follows the same scheme as in the *entities.json*
+- *surfaces* is a list of strings that refer to that unit. The library
     takes care of plurals, no need to specify them.
--   *entity* is the name of an entity in *entities.json*
--   *dimensions* follows the same schema as in *entities.json*, but the
+- *entity* is the name of an entity in *entities.json*
+- *dimensions* follows the same schema as in *entities.json*, but the
     *base* is the name of another unit, not of another entity.
--   *symbols* is a list of possible symbols and abbreviations for that
+- *symbols* is a list of possible symbols and abbreviations for that
     unit.
--   *prefixes* is an optional list. It can contain [Metric](https://en.wikipedia.org/wiki/Metric_prefix) and [Binary prefixes](https://en.wikipedia.org/wiki/Binary_prefix) and
+- *prefixes* is an optional list. It can contain [Metric](https://en.wikipedia.org/wiki/Metric_prefix) and [Binary prefixes](https://en.wikipedia.org/wiki/Binary_prefix) and
     automatically generates according units. If you want to
     add specifics (like different surfaces) you need to create an entry for that
     prefixes version on its own.
 
 All fields are case sensitive.
 
-Contributing
-------------
+### Contributing
+
 `dev` build: 
 
 [![Travis dev build state](https://travis-ci.com/nielstron/quantulum3.svg?branch=dev "Travis dev build state")](https://travis-ci.com/nielstron/quantulum3)
@@ -311,8 +329,8 @@ If you'd like to contribute follow these steps:
 (Optional, will be done automatically after pushing)
 8. Create a Pull Request when having commited and pushed your changes
 
-Language support
-----------------
+### Language support
+
 [![Travis dev build state](https://travis-ci.com/nielstron/quantulum3.svg?branch=language_support "Travis dev build state")](https://travis-ci.com/nielstron/quantulum3)
 [![Coverage Status](https://coveralls.io/repos/github/nielstron/quantulum3/badge.svg?branch=language_support)](https://coveralls.io/github/nielstron/quantulum3?branch=dev)
 
@@ -326,4 +344,3 @@ as in the automatic unittests.
 
 No changes outside the own language submodule folder (i.e. `_lang.de_DE`) should
 be necessary. If there are problems implementing a new language, don't hesitate to open an issue.
-

quantulum3/_lang/en_US/load.py

@@ -31,6 +31,7 @@ def number_to_words(number):
 ###############################################################################
 def build_common_words():
     # Read raw 4 letter file
+    units_ = load.units(lang)
     path = os.path.join(TOPDIR, "common-units.txt")
     with open(path, "r", encoding="utf-8") as file:
         common_units = {line.strip() for line in file if not line.startswith("#")}
@@ -42,15 +43,15 @@ def build_common_words():
                 continue
             line = line.rstrip()
             if (
-                line not in load.units(lang).surfaces_lower
-                and line not in load.units(lang).symbols
+                line not in units_.surfaces_lower
+                and line not in units_.symbols
                 and line not in common_units
             ):
                 words[len(line)].append(line)
             plural = load.pluralize(line)
             if (
-                plural not in load.units(lang).surfaces_all
-                and plural not in load.units(lang).symbols
+                plural not in units_.surfaces_all
+                and plural not in units_.symbols
                 and plural not in common_units
             ):
                 words[len(plural)].append(plural)

quantulum3/_lang/en_US/parser.py

@@ -242,6 +242,9 @@ def build_quantity(
     """
     # TODO rerun if change occurred
     # Re parse unit if a change occurred
+
+    units_ = load.units(lang)
+
     dimension_change = True
 
     # Extract "absolute " ...
@@ -250,7 +253,7 @@ def build_quantity(
         unit.name == "dimensionless"
         and _absolute == orig_text[span[0] - len(_absolute) : span[0]]
     ):
-        unit = load.units(lang).names["kelvin"]
+        unit = units_.names["kelvin"]
         unit.original_dimensions = unit.dimensions
         surface = _absolute + surface
         span = (span[0] - len(_absolute), span[1])
@@ -278,7 +281,7 @@ def build_quantity(
             # k/M etc is only applied if non-symbolic surfaces of other units
             # (because colloquial) or currency units
             symbolic = all(
-                dim["surface"] in load.units(lang).names[dim["base"]].symbols
+                dim["surface"] in units_.names[dim["base"]].symbols
                 for dim in unit.original_dimensions[1:]
             )
             if not symbolic:
@@ -430,7 +433,7 @@ def build_quantity(
                 unit.original_dimensions, orig_text, lang, classifier_path
             )
         else:
-            unit = load.units(lang).names["dimensionless"]
+            unit = units_.names["dimensionless"]
 
     # Discard irrelevant txt2float extractions, cardinal numbers, codes etc.
     if (