We’re excited to reveal a hotly anticipated development in how users of Ordnance Survey can access and automate product data downloads.
As part of our journey to make downloads more accessible, we’ve upgraded the scope of the OS Downloads API to include Premium data as well as OpenData. This compliments the existing offer to give users a complete & comprehensive list of products to download through the service. Before unpicking what this means, we’ll remind ourselves on the OS Downloads API service in general terms and what it’s useful for.
The service provides a mechanism for you to download your product of choice, without the need of user interaction such as manually downloading via web browser (on OS Data Hub) or requesting a hard copy version. The API service gives you the functionality to do the following:
- Discover which products (as data packages) are available to download
- Request metadata on data packages that are available to download
- Identify & request data packages that are for a specific area of interest (AOI)
- Identify & request data packages that for a specific format
- Once you’ve picked what to download, allow you to automate said download
This service works in conjunction with OS Data Hub so all a user would need to do as a prerequisite to using the API service, is to create a data package (or alter an existing data package) of their choosing on OS Data Hub (Get help with data packages on OS Data Hub).
With the Downloads API service explained, the remainder of this article will outline an example of where the service can be used to automate a premium data package download. There are various methods and tools available to do this and for this example we’ll use a ETL (extract, transform, load) software called FME to essentially ‘call’ the API service and build out a workflow to automate downloads.
Step 1
As our very first step, we will need to enable an instance of the OS Downloads API by navigating to the API page of OS Data Hub and adding the OS Downloads API service to a Project.
This will create an API key (given as a URL) that you can access/copy – make a note of this. It will be needed.
Still within OS Data Hub, it’s now a good idea to either build or pick an existing data package and take note of the Data Package ID that is generated. This can be found in grey text, just under the title of the Data Package in the Data Packages list window.
Finally, and still within the data package, take note of the Version ID that is generated. This can be found at the beginning of the file name of the individual file downloads.
Step 2
With this instance of the service now generated and a data package ready, we can now begin using FME to use call and automate download.
To begin, we will use the ‘Creator’ & ‘HTTPCaller’ transformers to initiate a call to the OS Downloads API service. Think of this as a placeholder that we are now asking the API to populate with information it has.
Specifically within the ‘HTTPCaller’ transformer parameters, we need to pay particular attention to the Request URL. This box can be populated with the API key that you kept hold of in Step 1.
As we also know the Data Package ID & Version ID, we can now begin modifying the URL in this box to include a specific data package & version that we have in mind.
To do this, we need to:
- Add the word ‘/dataPackages/’ after ‘v1’
- Then add the Data Package ID
- Then add the word ‘/versions/
- Then add the Version ID
- This should then continue with the API key as normal i.e. ?key=
(N.b. the image contains FME user parameters to input the information, so differs from the steps but the principles still apply.)
Step 3
Connect up a ‘JSONFragmenter’ transformer to the HTTPCaller transformer. The reason for this is because the response provided by the API is built into a JSON file. This transformer will allow us to split this file into the components we need to action a download.
In the parameters for the ‘JSONFragmenter’, try to match as the image and then execute the workflow. The result will provide a few columns for you to review. You are most interested in the ‘url’ column where you will be able to view and copy a specific url for the download of the file.
Step 4
Connect up a new ‘HTTPCaller’ transformer to the ‘JSONFragmenter’ transformer and in the ‘Request URL’ box of the parameters for the new ‘HTTPCaller’ transformer, add in the specific url that you were able to copy in Step 3.
You may at this stage want to configure the remaining part of the ‘HTTPCaller’ parameters including the ‘Output Folder’ box i.e. where you would like to save the file such as a folder location on your local hard drive.
Executing the workflow now will result in a download of your data package.
Step 5 – Optional
You may want to use the additional attributes that were exposed in Step 3. Doing a check on the ‘fileName’ and ‘size’ columns against the downloaded file you have will ensure the download has worked correctly and has not been corrupted in the process of downloading.