Go package implementing common properties and methods for working with Who's On First databases.
Documentation is incomplete at this time.
The package merges code from and supersedes the following packages:
- whosonfirst/go-whosonfirst-mysql
- whosonfirst/go-whosonfirst-database-sql
- whosonfirst/go-whosonfirst-database-sqlite
- whosonfirst/go-whosonfirst-opensearch
- whosonfirst/go-whosonfirst-elasticsearch
$> make cli
go build -mod vendor -ldflags="-s -w" -tags sqlite3 -o bin/wof-sql-create cmd/wof-sql-create/main.go
go build -mod vendor -ldflags="-s -w" -tags sqlite3 -o bin/wof-sql-index cmd/wof-sql-index/main.go
go build -mod vendor -ldflags="-s -w" -tags sqlite3 -o bin/wof-sql-prune cmd/wof-sql-prune/main.go
go build -mod vendor -ldflags="-s -w" -tags sqlite3 -o bin/wof-sql-iterwriter cmd/wof-sql-iterwriter/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-index cmd/wof-opensearch-index/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-query cmd/wof-opensearch-query/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-create-index cmd/wof-opensearch-create-index/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-delete-index cmd/wof-opensearch-delete-index/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-put-mapping cmd/wof-opensearch-put-mapping/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-get-mapping cmd/wof-opensearch-get-mapping/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-put-settings cmd/wof-opensearch-put-settings/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-list-indices cmd/wof-opensearch-list-indices/main.go
go build -mod vendor -ldflags="-s -w" -o bin/wof-opensearch-indices-stats cmd/wof-opensearch-indices-stats/main.go
Database support is enabled through tags. The following tags are supported:
| Tag | Package | Notes |
|---|---|---|
| sqlite3 | mattn/go-sqlite3 | |
| mysql | go-sql-driver/mysql | |
| postgres | lib/pq |
The default cli Makefile target will build tools with the sqlite3 tag. If, for example, you wanted to building the database indexing tool with support for MySQL you would do this:
go build -build mysql -mod vendor -ldflags="-s -w" -tags sqlite3 -o bin/wof-sql-index cmd/wof-sql-index/main.go
Support for MySQL and Postgres database should still be considered experimental. Most of the development to date has centered around SQLite so there will almost certainly be "gotchas", maybe even bugs, with other database engines.
Create, but do not index, one or more tables in a database/sql compatiable database.
$> ./bin/wof-sql-create -h
-all
Index all tables (except the 'search' and 'geometries' tables which you need to specify explicitly)
-ancestors
Index the 'ancestors' tables
-concordances
Index the 'concordances' tables
-database-uri string
...
-geojson
Index the 'geojson' table
-geometries
Index the 'geometries' table (requires that libspatialite already be installed)
-names
Index the 'names' table
-properties
Index the 'properties' table
-rtree
Index the 'rtree' table
-search
Index the 'search' table (using SQLite FTS5 full-text indexer)
-spatial-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spatial-sqlite package.
-spelunker
Index the 'spelunker' table
-spelunker-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spelunker packages
-spr
Index the 'spr' table
-supersedes
Index the 'supersedes' table
-verbose
Enable verbose (debug) logging
Index one or more tables in a database/sql compatiable database.
$> ./bin/wof-sql-index -h
-all
Index all tables (except the 'search' and 'geometries' tables which you need to specify explicitly)
-ancestors
Index the 'ancestors' tables
-concordances
Index the 'concordances' tables
-database-uri string
A URI in the form of 'sql://{DATABASE_SQL_ENGINE}?dsn={DATABASE_SQL_DSN}'. For example: sql://sqlite3?dsn=test.db
-geojson
Index the 'geojson' table
-geometries
Index the 'geometries' table (requires that libspatialite already be installed)
-index-alt value
Zero or more table names where alt geometry files should be indexed.
-index-relations
Index the records related to a feature, specifically wof:belongsto, wof:depicts and wof:involves. Alt files for relations are not indexed at this time.
-index-relations-reader-uri string
A valid go-reader.Reader URI from which to read data for a relations candidate.
-iterator-uri string
A valid whosonfirst/go-whosonfirst-iterate/v3.Iterator URI. Supported iterator URI schemes are: cwd://,directory://,featurecollection://,file://,filelist://,geojsonl://,null://,repo:// (default "repo://")
-names
Index the 'names' table
-optimize
Attempt to optimize the database before closing connection (default true)
-processes int
The number of concurrent processes to index data with (default 20)
-properties
Index the 'properties' table
-rtree
Index the 'rtree' table
-search
Index the 'search' table (using SQLite FTS5 full-text indexer)
-spatial-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spatial-sqlite package.
-spelunker
Index the 'spelunker' table
-spelunker-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spelunker packages
-spr
Index the 'spr' table
-strict-alt-files
Be strict when indexing alt geometries (default true)
-supersedes
Index the 'supersedes' table
-timings
Display timings during and after indexing
-verbose
Enable verbose (debug) logging
$> ./bin/wof-sql-index \
-spatial-tables \
-timings \
-database-uri 'sql://sqlite3?dsn=test2.db' \
/usr/local/data/sfomuseum-data-whosonfirst
2025/09/12 12:46:31 INFO Iterator stats elapsed=27.444911792s seen=1604 allocated="1.6 MB" "total allocated"="10 GB" sys="284 MB" numgc=2650
And then to use that database with, for example, the whosonfirst/go-whosonfirst-spatial-sqlite` package:
$> cd /usr/local/whosonfirst/go-whosonfirst-spatial-sqlite`](#) package:
$> ./bin/pip \
-spatial-database-uri 'sqlite://sqlite3?dsn=/usr/local/whosonfirst/go-whosonfirst-database/test2.db' \
-latitude 37.616951 \
-longitude -122.383747 \
| jq -r '.places[]["wof:name"]'
Earth
North America
United States
California
San Mateo
San Francisco International Airport
94128
Remove all the records from one or more tables in a database/sql compatible Who's On First database.
$> ./bin/wof-sql-prune -h
-all
Index all tables (except the 'search' and 'geometries' tables which you need to specify explicitly)
-ancestors
Index the 'ancestors' tables
-concordances
Index the 'concordances' tables
-database-uri string
...
-geojson
Index the 'geojson' table
-geometries
Index the 'geometries' table (requires that libspatialite already be installed)
-names
Index the 'names' table
-properties
Index the 'properties' table
-rtree
Index the 'rtree' table
-search
Index the 'search' table (using SQLite FTS5 full-text indexer)
-spatial-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spatial-sqlite package.
-spelunker
Index the 'spelunker' table
-spelunker-tables
If true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spelunker packages
-spr
Index the 'spr' table
-supersedes
Index the 'supersedes' table
-verbose
Enable verbose (debug) logging
$> ./bin/wof-opensearch-index \
-writer-uri 'constant://?val=opensearch2%3A%2F%2Flocalhost%3A9200%2Fspelunker%3Frequire-tls%3Dtrue%26insecure%3Dtrue%26debug%3Dfalse%26username%3Dadmin%26password%3Ds33kret' \
/usr/local/data/whosonfirst-data-admin-xy
2024/03/09 22:36:53 time to index paths (1) 1.530319599s
2024/03/09 22:36:57 INFO Index complete indexed=18432
Note the unfortunate need to URL escape the -writer-uri=constant://?val= parameter which unescaped is the actual go-whosonfirst-opensearch/writer.OpensearchV2Writer URI that takes the form of:
opensearch2://localhost:9200/spelunker?require-tls=true&insecure=true&debug=false&username=admin&password=s33kret
The wof-opensearch-index application however expects a gocloud.dev/runtimevar URI so that you don't need to deply production configuration values with sensitive values (like OpenSearch admin passwords) exposed in them. Under the hood the wof-opensearch-index application is using the sfomuseum/runtimevar package to manage the details and this needs to be updated to allow plain (non-runtimevar) strings. Or maybe the wof-opensearch-index application needs to be updated. Either way something needs to be updated to avoid the hassle of always needing to URL-escape things.
Create a new OpenSearch index with optional settings and mappings.
$> ./bin/wof-opensearch-create-index \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
-opensearch-index collection \
-settings /usr/local/sfomuseum/es-sfomuseum-schema/schema/7.4/mappings.collection.json
{"acknowledged":true,"shards_acknowledged":true}
Delete an OpenSearch index.
$> ./bin/wof-opensearch-delete-index \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
-opensearch-index collection \
Index one or more whosonfirst/go-whosonfirst-iterate/v2 sources in an OpenSearch index.
$> ./bin/wof-opensearch-index \
-writer-uri 'constant://?val=...' \
/usr/local/data/sfomuseum-data-collection-classifications/
2023/09/15 23:42:19 time to index paths (1) 10.551282589s
List all the indices for an OpenSearch instance.
$> ./bin/wof-opensearch-list-indices \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
Retrieve the mappings for an OpenSearch index.
$> ./bin/wof-opensearch-get-mapping \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
-opensearch-index collection
Assign and update new mappings in an OpenSearch index. Remember: It is not possible to change existing mappings in an index.
$> ./bin/wof-opensearch-put-mapping \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
-opensearch-index collection \
-mapping mapping.json
Query an OpenSearch index.
$> ./bin/wof-opensearch-query \
-opensearch-aws-credentials-uri 'aws://{REGION}?credentials=iam:' \
-opensearch-endpoint https://{OPENSEARCH_DOMAIN}.{REGION}.es.amazonaws.com \
-opensearch-index collection \
'{"query": { "ids": { "values": [ 1880245177 ] } } }' | jq
{
"took": 914,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "collection",
"_id": "1880245177",
"_score": 1,
"_source": {
"counts:names_languages": 0,
"counts:names_prefered": 0,
"counts:names_total": 0,
"counts:names_variant": 0,
"edtf:cessation": "",
"edtf:inception": "",
"geom:area": 0,
"geom:bbox": "-122.386155,37.616357,-122.386155,37.616357",
"geom:latitude": 37.616357,
"geom:longitude": -122.386155,
"mz:is_current": 1,
"sfomuseum:category": "Scheduling / Ticketing",
"sfomuseum:classification_id": -1,
"sfomuseum:collection": "Aviation Museum",
"sfomuseum:placetype": "subcategory",
"sfomuseum:subcategory": "Luggage Tag, Crew",
"src:geom": "sfomuseum",
"translations": [],
"wof:belongsto": [
1511214277,
102527513,
1762679689,
1511214203,
102191575,
85633793,
102087579,
85922583,
85688637
],
"wof:country": "US",
"wof:created": 1693337008,
"wof:geomhash": "30c8a918561c84bb2daa2b97fc7c5353",
"wof:hierarchy": [
{
"building_id": 1511214277,
"campus_id": 102527513,
"category_id": 1762679689,
"collection_id": 1511214203,
"continent_id": 102191575,
"country_id": 85633793,
"county_id": 102087579,
"locality_id": 85922583,
"neighbourhood_id": -1,
"region_id": 85688637,
"subcategory_id": 1880245177,
"wing_id": 1511214203
}
],
"wof:id": 1880245177,
"wof:lastmodified": 1693337008,
"wof:name": "Luggage Tag, Crew",
"wof:parent_id": 1762679689,
"wof:placetype": "custom",
"wof:placetype_alt": "subcategory",
"wof:placetype_id": 1729783759,
"wof:placetype_names": [
"custom",
"subcategory"
],
"wof:repo": "sfomuseum-data-collection-classfications",
"wof:superseded_by": [],
"wof:supersedes": []
}
}
]
}
}