2. Data Collection, Querying and Retrieval: TelemetryStore

The TelemetryCollector is responsible for collecting, querying and storing the data received from Azure Application Insights. It consists of the following parts:

• DataCollection: Where data is collected from Application Insights
• Model: The data model used to store the data
• Store: The logic used to store the data
• Filter: Where data can be queried by the user
• Persistance: A persistance module to temporarily store the collected data on disk

Data Collection

Data Collection is designed to work with the Azure Application Insights (AAI) External REST API interface. It allows a user to query telemetry data of a certain AAI instance. It consists of a IDataCollection interface that has to be implemented by any new added data collection method. The used data collection method is defined in DataCollectionServiceProvider, it is also possible to use multiple sources at a time. The arriving data is stored in an object called CollectedDataEntity, in the Azure REST API case a data container that stores all relevant data. It contains conversion methods to return Store Objects that can actually be stored (a CollectedDataEntity cannot be stored, it has to be converted).

Model

The Model is used to store and distribute that data. It therefore contains two different kinds of elements: ConcreteMethod, of which an instance is created for every incoming data telemetry element, as well as AveragedMethod, which summarizes ConcreteMethod elements on a method level. ConcreteMethod elements can be filtered on their properties. All stores therefore consists of a collection of AveragedMethod elements, one per method, and a collection of ConcreteMethod elements per method and per store. ConcreteMethod has to be subclassed to create a new kind of store. The current implementation provides a ConcreteMethodTelemetry container to store "normal" telemetry data.

Store

The logic for storing the actual data resides here. Store is type generic, the structure of the data to be stored is provided by the model (in our case ConcreteMethodTelemetry or ConcreteMethodException). New stores are instantiated in the StoreManager class that contains logic to fill new data into the stores, as well as generating summary objects (AveragedMethod). A store contains lists of all telemetry elements (in our case: ConcreteMethodTelemetry), as well as a list of those elements that were not rejected by any filter. It supplies a FilterController that can be used over an interface in VS, as well as a PersistanceService that is used to store the data on disk. To be as responsive as possible, the telemetry elements are stored in nested ConcurrentDictionaries, where the first layer is the method name where the telemetry element belongs to and the second layer the AAI id provided by Azure.

The StoreManager contains the logic necessary to bring the stores, the data model as well as the data collection together. Data Collection is triggered by a Background Task that calls method run, and then recalls itself after a specified amount of time. If updates occurred since the last collection of data, the stores are updated asynchronously. The AveragedMethod instance is stored here and contains summarized information of all used stores.

Filter

Filter are applied on a store level basis and are also type generic. Inside a store, there exist two ConcurrentDictionaries: AllMemberTelemetries and CurrentMemberTelemetries. AllMemberTelemetries minus filteredObjects equals CurrentMemberTelemetries. Detection of properties that can be filtered is done automatically by reflection. When a new FilterController is instantiated, it also receives the type generics of the data and creates FilterProperties based on all properties available in the provided type. These properties can be retrieved by an interface in VS and used to create new filters. Currently, filters are implemented for following data types:

• DateTime
• String
• Int32

Filters can be applied on one method only, or globally. They need to be applied for multiple stores manually.

Persistance

The persistance module on one hand serializes the model and stores it in a temporary folder on disk, on the other provides the WritableSettingsStoreController which allows storage of access information for AAI into the computers registry. Should VS be closed and reopened, the data is retrieved from the file system.