2. Data Collection, Querying and Retrieval: TelemetryStore

The TelemetryCollector is responsible for collecting, querying and storing the data received from Azure Application Insights. It consists of the following parts:

• DataPulling: Where data is collected from Application Insights
• Model: The data model used to store the data
• Store: The logic used to store the data
• Filter: Where data can be queried by the user
• Persistance: A persistance module to temporarily store the collected data on disk

Data Pulling

Data Pulling is designed to work with the Azure Application Insights (AAI) External REST API interface. It allows a user to query telemetry data of a certain AAI instance. It consists of a IDataPullingService interface that has to be implemented by any new added data collection method. The used data pulling method is defined in DataPullingServiceProvider, it is also possible to use multiple sources at a time. The arriving data is stored in an object called PulledDataEntity, in the Azure REST API case a data container that stores all relevant data. It contains conversion methods to return Store Objects that can actually be stored (a PulledDataEntity cannot be stored, it has to be converted).

Model

The Model is used to store and distribute that data. It therefore contains two different kinds of elements: ConcreteMethod, of which an instance is created for every incoming data telemetry element, as well as AveragedMethod, which summarizes ConcreteMethod elements on a method level. ConcreteMethod elements can be filtered on their properties. All stores therefore consists of a collection of AveragedMethod elements, one per method, and a collection of ConcreteMethod elements per method and per store. ConcreteMethod has to be subclassed to create a new kind of store. The current implementation provides a ConcreteMethodTelemetry container to store "normal" telemetry data.

Store

The logic for storing the actual data resides here. Store is type generic, the structure of the data to be stored is provided by the model (in our case ConcreteMethodTelemetry). The class StoreProvider contains instantiations of all used stores, as well as a collection over all stores. The collection GetStores() can only be used to update the stores, but not to query any information from them. A store contains lists of all telemetry elements (in our case: ConcreteMethodTelemetry), as well as a list of those elements that were not rejected by any filter. It supplies a FilterController that can be used over an interface in VS, as well as a PersistanceService that is used to store the data on disk. To be as responsive as possible, the telemetry elements are stored in nested ConcurrentDictionaries, where the first layer is the method name where the telemetry element belongs to and the second layer the AAI id provided by Azure.

The StoreHandler contains the logic necessary to bring the stores, the data model as well as the data collection together. Data Pulling is triggered by a timer that is executed every 5 seconds by the method RunPipeline. If updates occurred since the last pull of data, the stores are updated in the background. The AveragedMethod instance is stored here and contains information of all used stores.

Filter

Filter are applied on a store level basis and are also type generic. Inside a store, there exist two ConcurrentDictionaries: AllMemberTelemetries and CurrentMemberTelemetries. AllMemberTelemetries minus filteredObjects equals CurrentMemberTelemetries. Detection of properties that can be filtered is done automatically by reflection. When a new FilterController is instantiated, it also receives the type generics of the data and creates FilterProperties based on all properties available in the provided type. These properties can be retrieved by an interface in VS and used to create new filters. Currently, filters are implemented for following data types:

• DateTime
• String
• Int32 Filters can be applied on one method only, or globally.

Persistance

The persistance module on one hand serializes the model and stores it in a temporary folder on disk, on the other provides the WritableSettingsStoreController which allows storage of access information for AAI into the computers registry. Should VS be closed and reopened, the data is retrieved from the file system.