Tracking changes in product data and how to use it for caching
In this part we want to answer two questions. How can you detect that product data has changed in OKAPI and how you can use this information to update your cached product data? There are different cases where you cache product data on your side. We won't explicitly go into those but will explain in general how we provide ways to keep product data up to date on your side and how we signal that product data has been updated.
OKAPI is a dynamic product data API where product data can change at any time. To detect changes, we have provided several options. Each of our catalog endpoints respond with an ETag and an X-(Country, Brand, Model, Type)-Content-Lastupdate(-Long) timestamp (in a human and machine-readable format). Only the header of /countries and /languages endpoints do not include an X-Catalog-Content-Lastupdate.
Response header:
| X-Catalog-Content-Lastupdate | X-Catalog-Content-Lastupdate-Long | ETag |
|---|---|---|
| Value: Timestamp UTC | Value: Unix timestamp | Value: Unique Hashcode |
| Identifies catalog product data with value | Identifies catalog product data with value | Identifies an OKAPI response with value |
| Example: X-Model-Content-Lastupdate: Fri, 31 Jul 2020 02:41:18 UTC |
Example: X-Model-Content-Lastupdate-Long: 1596156078 |
Example: Etag: e3c780bf6e4a11b8 |
ETag: What is it and how to use it
The ETag, included in the response header, is a calculated hash value based on a specific OKAPI response at the endpoint. An ETag changes only when the content of the response at the endpoint changes. Specifying "if-none-match" with the ETag hash value in the request header will check if the ETag changes. If the content of the response and the ETag hash value do not change, OKAPI responds with an http status 304 "not modified".
Example: Use ETag for models endpoint
The ETag hash value at “v3/catalog/DE/brands/VW/models” endpoint only changes when the included product data changes, due to modification. Examples would be a model is no longer included, a new model is added, the description of a model changes, or any combination of the previous. The following table shows the events, which can happen when calling the endpoint and using and not using ETag.
| Event | http Status | OKAPI Response |
|---|---|---|
| Call v3/catalog/DE/brands/VW/models without added “if-none-match” in header | 200 “OK” | Valid OKAPI response with ETag: e3c780bf6e4a11b8 in Header |
| Call v3/catalog/DE/brands/VW/models with added “if-none-match= e3c780bf6e4a11b8” in header | 304 “not modified” | Empty response body |
| Call v3/catalog/DE/brands/VW/models with added “if-none-match= e3c780bf6e4a11b8” in header | 200 “OK” | Valid OKAPI response with changes in response and new ETag: e3c780bf6e41a1s6 in header |
ETag: Possible caching concept
For example, if you often need the answer from “v3/catalog/DE/brands/VW/models” endpoint but want to cache the answer at your end and not query it all the time, you can use the ETag hash value contained in the answer and specify it in the request header using "if-none-match". If you receive a response with http status 304 "not modified", the response of the endpoint has not changed. Only with a http status 200 "OK" in the response you must update your cache.
X-Catalog-Content-Lastupdate(-Long): What is it and how to use it
We constantly keep our product data up to date. In addition to daily rebuilding of the product data, the affected product data is also rebuilt when individual markets are updated during the day. Despite daily rebuilding of the product data, the content of the product data for a market does not necessarily have to change. To track changes to the content of the product data, we provide an X-Catalog-Content-Lastupdate(-Long) timestamp at the catalog endpoints. We currently provide the last update timestamp in two formats, using Coordinated Universal Time (UTC).
| Header entry | Format | Time |
|---|---|---|
| X-Catalog-Content-Lastupdate | Human readable | Wed, 12 Jan 2022 03:54:24 UTC |
| X-Catalog-Content-Lastupdate-Long | Unix timestamp in milliseconds | 1641959664604 |
We determine the last update timestamp for each country, brand, model, and type, which are arranged hierarchically at different levels. The X-Type-Content-Lastupdate describes the lowest level and the X-Country-Content-Lastupdate the highest. The higher the level, the more likely are changes to the content of the product data and the update of the last update timestamp. Below is an explanation of when the last update timestamp changes.
X-Type-Content-Lastupdate(-Long): Changes to the timestamp only occur when a field in the type object or something in the associated options changes. For example, an option is no longer valid and has been removed or the description of an option has changed. It also means that the associated timestamps for country, brand, and model change.
X-Model-Content-Lastupdate(-Long): Timestamp changes only occur when a field in the model object or something in the associated types and options changes. In addition, the associated timestamps of country and brand are also updated.
X-Brand-Content-Lastupdate(-Long): A change of the timestamp only occurs when a field or object is changed in the associated models, types, or options. Changing this timestamp also affects the timestamp of the associated country.
X-Country-Content-Lastupdate(-Long): This changes most often when a field or object is changed in the associated brands, models, types, or options.
Example: Data update for market VW Germany
| Level | Deleted | Created | Updated | Unchanged |
|---|---|---|---|---|
| Country | 0 | 0 | 1 | 0 |
| Brand | 0 | 0 | 1 | 0 |
| Model | 1 | 1 | 3 | 22 |
| Type | 44 | 42 | 86 | 325 |
In this example, changes have occurred for several models and types. One model has been removed, one has been added, and the underlying content has changed for three models. Other models and types have not been adjusted. In this case the timestamps for the given brand and country changed caused by the changes in the lower level.
X-Catalog-Content-Lastupdate(-long): Possible caching concept
If you want to cache product data for a hole market such as VW Germany, the X-Brand- Content-Lastupdate Timestamp is the ideal indicator to update the product data in your cache. For this purpose, we currently offer the possibility to query the endpoint v3/catalog/DE/brands/VW in regular intervals and to check the X-Brand-Content-Lastupdate or X-Brand-Content-Lastupdate-long Timestamp in the header if it differs from the last data update received last update timestamp. If this is the case, you should rebuild your cache. With the last update timestamps for models and types we offer you further possibilities to refine your cache concept to update only the product data that really needs to be updated.
How ETag and X-Catalog-Content-Lastupdate(-Long) timestamp working together
The ETag and the X-Catalog-Content-Lastupdate(-Long) timestamp are related, but not necessarily dependent on each other. We'll give you an example. If you execute the same request again after a certain time interval, one of the following cases may occur:
Case 1: New data in OKAPI for the requested endpoint (ETag change + timestamp change)
There is a different response because the data has been updated. Both headers will be affected.
Case 2: New data in OKAPI but not listed in the requested endpoint (ETag same + timestamp change)
The ETag header remains the same because the content of the response from OKAPI does not change. The updated product data is not visible in the specified endpoint. But the timestamp changes because something has changed in the product data present under the timestamp.
Case 3: : No new data in OKAPI (ETag same + timestamp same)
The ETag header and the last update timestamp did not change.
Case 4: Using a query parameter (ETag change + timestamp same)
When catalog endpoints are requested with a query parameter, the OKAPI response changes, but not the catalog data.