Developer Notes

API overview

The Operational Data Sharing (ODS) defines an HTTP API and data structure for a JSON-formatted data product. The API specification is documented using the OpenAPI schema and is accessible at https://obs.vla.nrao.edu/ods/openapi.json. More about the OpenAPI initiative can be found at https://www.openapis.org.

The ODS API can be realized through various architectural approaches. The following is an illustrative example inspired by the NRAO's implementation.

The ODS data is saved in a dedicated database (DB). Multiple database technologies allow saving JSON as a single DB field and perform queries on its keys. For example, here is some of the JSON capabilities for PostgreSQL (version 12 onward): https://www.postgresql.org/docs/current/functions-json.html.

The API acts as a wrapper for writing and reading from the database. It can be created in multiple languages, such as JavaScript or Python. For example, https://github.com/python-openapi/openapi-core is a Python library that adds client-side and server-side support for OpenAPI.

An important feature is ensuring that the published data is compliant with the OpenAPI schema. This can be ensured by the Data sender, by the API before writing to the database, or by the API before publishing to the consumer. Again, OpenAPI offers ways to validate JSON data against the ODS schema.

Endpoint examples

The OpenAPI standard currently only specifies the response for the http://{host}:{port}/ods_data endpoint (or "path"), with no arguments. As an example, we assume that the API runs on port 8000. A GET request to http://host:8000/ods_data/ must return the expected data (a JSON list for current or starting observations. See the "Data anticipation" section).

The use of arguments or other endpoints can be customized. For development or debugging, it is interesting to retrieve what has been published in ODS in a certain period of time. This could be done by passing arguments to the API, e.g.: http://host:8000/ods_data/?from='2024-11-01 15:00:00'&to='2024-11-01 15:20:00'. This time range can refer to the API write timestamp, or to the src_start_utc JSON key. This avoids direct access to the database.

We recommend hiding the API port and restricting external access only to the endpoints specified in the schema (e.g., using a forward proxy such as the Apache-httpd).

Data anticipation

Observations can be published in the ODS API once they have been scheduled for the telescope or array. This can happen many hours in advance. However, observatories working with dynamic scheduling (such as the VLA) can have their schedule greatly altered due to changes in weather conditions.

To minimize the publication of "false positives" in the API (i.e., observations that don't happen), the VLA limits itself to publishing the most recent version of its schedule 20 minutes in advance.

Changelog

An architecture for authenticating data consumers and producers for the API is currently under discussion. Also, a way of reporting long periods of inactivity at the observatory (e.g., maintenance periods).

The version of the OpenAPI specification will be incremented whenever it changes. The initial version was called "v.helloworld.1". A history of the schema files will be kept on the main ODS page.

About

Contact: Bang Nhan (NRAO)

Acknowledgment

The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc. The ODS system was developed by NRAO with support from the National Science Foundation's SII NRDZ: Dynamic Protection and Spectrum Monitoring for Radio Observatories (AST-2232159), and the SWIFT-SAT: Observational Data Sharing (AST-2332422).