Multi-tenant custom field support

[ejsmith]

Patternize multi-tenant custom field handling

• Custom field mapping storage
  • Entity type (repo), tenant (should allow for composite like organization+type), field type, friendly name, extensible to store additional mapping information
  • Shared across multiple entity types and tenants
  • Auto assign index slots to new custom fields
  • Fill in gaps for index slots when fields have been deleted
  • Ability to efficiently lookup field mapping for an entity type and tenant
    • Aggressively cache mapping for the tenant
  • Ability to have a list of aliases for a field that will be used to resolve queries
    • Only the root name will be used for getting data that gets added to the idx slot
    • Aliases count as used names can't be used by other fields
• Extensible field types
  • Packaged up field type behaviors that can be re-used
  • Built in field types for string, date, bool, number, float, etc
  • Consumers can create custom field types to add additional indexing formats like address, geo, etc
  • Ability to transform values when adding them to idx
    • Allows the data to be optimized for searching
    • Allows things to happen like looking up related data to store denormalized
    • Probably needs to allow batching of values for efficiency when doing idx transforms
  • Standard field types automatically configured, but can be overridden with custom behavior
  • Ignore errors for invalid document field values when saving
  • Ability to do advanced things like scripted fields that have access to the rest of the document to calculate values (proven out in tests) https://github.com/FoundatioFx/Foundatio.Repositories/blob/master/tests/Foundatio.Repositories.Elasticsearch.Tests/Repositories/Configuration/Indexes/CalculatedIntegerFieldType.cs
• Easily implemented in repositories
  • Interface for model to implement that opts the repository into supporting custom fields
  • Automatically copies values from friendly names to idx slots
  • Automatically gets mapping and sets QueryFieldResolver on queries
• Field management
  • Ability to change type of an existing field
    • Could start out as an auto field indexed as string and then be changed to a date or some other more specific type
  • Ability to remove a field
    • Cleanup idx values
    • Optionally remove custom field values
  • Ability to rename a field
• Ability to auto create custom fields on the fly when we see a new value that does not have a mapping
  • Default to store as string with keyword subfield
  • Hook when auto creating custom property mappings that lets you decide if a mapping should be created and determine the default index type

[ejsmith]

Changing data type flow:

• Soft delete existing field
  • Name will be freed up to be used
  • Slot will remain unavailable until fully deleted
  • Data in old idx slot will still be there until we reprocess, but it won't be queried because there won't be a mapping pointing to it any more (it's soft deleted)
• Create new field with new index type and same name, will get a new slot for that index type
  • Copy all settings from previous custom field like the extended data and description, etc so it is a full copy
  • Any new documents will get the field saved to the correct slot
  • Any saved documents will get the idx values reconfigured with the new type
• Batch process any documents where the previous idx slot has a value and just re-save the documents
  • What happens if some documents have incompatible custom field values?
• Fully delete old custom field (no need to keep it around longer)

[ejsmith]

Remove field flow:

• Soft delete field
  • Name is available to be used, if we try to undelete someone might have taken the name
  • Field map will be immediately removed and won't be able to search the field
  • Orphaned idx data needs to be cleaned up before we allow the slot to be re-used
• Batch process any documents where the idx slot has a value and just re-save the documents
• Optionally delete the values from the custom field source as well (not idx)
  • Need new method on IHaveCustomFields to delete a custom field value
  • Seems valid to want to leave the data, but have it be unmapped
• Fully delete old custom field

[ejsmith]

Rename field flow:

• Change name on custom field
  • Add old name as an alias
  • Name won't be allowed to be used since it's being used as an alias
  • Auto create won't attempt to create mapping because the name is used as an alias
• Batch process any documents where the idx slot has a value and move data from old name to new name
• Remove old name from alias list which will open up the name to be used again

[ejsmith]

Make sure that we have everything in place that we would need to support calculated field custom properties. Should have access to the field definition and access to the full document. Also, we would need to calculate the display value as well as the idx value.

[niemyjski]

Is this up to date?

[ejsmith]

Yeah, I think that's where I left off.