1   Introduction¶

The current implementation of the Engineering and Facilities Database (EFD) is based on InfluxDB, an open source time series platform optimized for efficient storage and analysis of time series data.

While the Summit EFD instance is used for real-time monitoring of the Observatory, the raw data stream (Telemetry, Events, and Commands) is also replicated to the LSST Data Facility (LDF), and accessible to the project at large via the LDF EFD instance.

In this technote, we revisit the EFD transformation service, initially described in DMTN-050. The main goal of the EFD transformation service is to curate the EFD telemetry data, relate it to the observations, and publish it to the Consolidated Database enabling access to the Science Platform science users.

In particular, we review the EFD architecture and the components of the EFD transformation service. We analyze the schemas of telemetry topics from different Observatory subsystems and suggest an approach for creating transformed telemetry tables in the Consolidated Database.

We show how the Kafka Connect JDBC Sink connector can create, evolve the schema, and write to the transformed telemetry tables consuming the transformed telemetry streams.

Finally, we discuss how to extend the kafka-aggregator tool to produce the transformed telemetry streams.

2   Architecture¶

Figure 1 shows the components of the EFD transformation service at LDF.

Figure 1 Components of the EFD transformation service

3   EFD telemetry topics¶

May 10, 2021, ts_xml defines the schema for 249 Telemetry, 390 Commands, and 533 Events topics for 62 different Observatory subsystems.

SAL translates the ts_xml schemas for each topic into IDL schemas used by DDS. The IDL schemas are then translated to Avro schemas and registered in the EFD when the SAL Kafka producers are initialized.

In this investigation, we assume that the leading interest of Science Platform science users is the EFD Telemetry data, and that Events and Commands do not need to be recorded into the Consolidated Database (see section 8   Discussion).

In this section, we analyze the schemas for several Observatory subsystems and use the WeatherStation subsystem as an example.

4   Transformed telemetry tables¶

Let’s use the WeatherStation telemetry topics to examplify the creation of a transformed telemetry table.

• The transformed WeatherStation telemetry table combines information from multiple WeatherStation telemetry topics.
• Fields that are not relevant to the Science Platform science user are excluded. In particular, most of the private_ fields added by SAL can be excluded and others reduced to a single timestamp field.
• In this particular example, the original topics have aggregated fields like min24H, avg24H, max24H. We decided to keep only the fields with “1 minute average values”, which are available in most of the cases, and leave it up to the user to compute aggregations in SQL as needed.
• Field names are namespaced to identify the original EFD topic.

From this example, we conclude that to create a transformed telemetry table, the EFD transformation service must be able to specify a mapping between the source telemetry topics and the transformed telemetry table, and specify which fields within those topics to use. In some cases, it must be able to apply transformations to the fields’ values, and allows for new descriptions and units for the transformed fields.

In other words, the EFD transformation service holds the transformations (decisions) necessary to create the Consolidated Database telemetry tables from the raw EFD telemetry topics.

5   The Kafka Connect JDBC Sink connector¶

In this section, we describe some features of the Kafka Connect JDBC Sink connector and how it is used in the EFD transformation service.

'transforms'                          = 'flatten',
'transforms.flatten.type'             = 'org.apache.kafka.connect.transforms.Flatten$Value'

'transforms.setTimestampType.type'        = 'org.apache.kafka.connect.transforms.TimestampConverter$Value',
'transforms.setTimestampType.field'       = 'timestamp',
'transforms.setTimestampType.target.type' = 'Timestamp'

'pk.mode'                                 = 'record_value',
'pk.fields'                               = 'timestamp',

6   Transformed telemetry streams¶

A table is the materialization of a stream. In the previous section, we showed how the JDBC Sink connector can be used to create the transformed telemetry tables.

In this section, we discuss how to extend the kafka-aggregator tool to produce the transformed telemetry streams.

---
aggregated_topic_name1:
   mapping:
      source_topic_name1:
         field1:
            name: new_name
            description: "new description for the transformed field"
            units: "new units for the transformed field"
            transformation: func1
         field2:
            description: "new description for the transformed field"
            units: "new units for the transformed field"
            transformation: func2
         field3:
         ...
      source_topic_name2:
         field1:
         field2:
         field3:
         ...
      ...
aggregated_topic_name2:
   window_aggregation_size: 1s
   operations:
      - min
      - median
      - max
   mapping:
      source_topic_name3:
         field1:
         field2:
         field3:
         ...
      ...
...

7   Relating Telemetry data with the observations¶

8   Discussion¶

Why publishing only EFD telemetry data to the Consolidated Database?

The EFD data comprises telemetry, Events, and Commands topics. While Events and Commands are crucial for engineers in understanding the telescope systems during operations, they are less critical to science users. Telemetry is essential for science users to correlate with data quality after data acquisition and data processing.

What happens if the science user needs data from the EFD that is not published to the Consolidated Database?

That is a common problem of designing a schema upfront and perhaps the most sensitive aspect of EFD transformation service.

The proposed solution is flexible enough to allow changes to the EFD Consolidated Database schema that are forward compatible. It is possible to add new tables and columns to existing tables in the Consolidated Database at any given time. The forward compatibility of the EFD Consolidated Database schema ensures that queries that worked with the old schema continue to work with the new schema. Similarly, queries designed to work with the new schema only return meaningful values for data inserted after the schema change.

The above may represent a limitation for the current solution because the proposed process will not perform a batch load of the historical EFD data when the Consolidated Database schema changes. Replay the raw EFD data from Parquet files to Kafka might be an option, but it is out of the scope of this implementation.

Is the EFD transformation service also responsible for creating “Exposure tables” for the AT and the MT in the Consolidated Database?

DMTN-050 mentions relations between the telemetry tables and Exposure tables, but it is not clear who is responsible for creating the latter.

In principle, the ATExposure and MTExposure tables in the Consolidated Database can be derived from the ATCamera_logevent_endReadout and MTCamera_logevent_endReadout Events. When these events are received, the corresponding images should be complete.