Running batch jobs

Batch jobs provide an efficient way to execute bulk operations, including:

• Downloading property data for up to 1 million properties (see Property details jobs.
• Obtaining property IDs for up to 100,000 properties by address (see Match jobs).
• Obtaining property IDs for large numbers of properties that match the specified search criteria (see Search jobs).

These batch job types (property-details, match, and search) use the POST /v2/jobs/file endpoint.

Batch job workflow

Batch jobs run asynchronously, meaning you submit your request and must wait until the operation completes before you can download the results. The steps to do this are:

1. Create a batch job specifying the job type (property-details or match) and the job details:

   Job type	Job details
   property-details	Specify the type of data you want (for example, ownership) and list of property IDs (up to 1 million).
   match	Provide a CSV file with a list of property addresses or lat/lon coordinates (up to 100,000).

2. Get the job ID from the initial API response.

3. Poll the API to determine when the job is complete.

4. Download the resulting data using the URL(s) returned by the API:

   Job type	Returns
   property-details	The property data for the requested properties.
   match	Your CSV file with the matching Reonomy property ID appended to each row.

For property details jobs ("job_type": "property-details"), you must include a job spec file specifying a list of property IDs and the type of information you're seeking (the detail_type). You can include multiple types in a single request. Valid detail types are:

• basic
• mortgages
• ownership
• reported_owner
• sales
• taxes
• tenants

You can upload the job specification in two ways:

• By referencing a file on your local machine using the file keyword in the request (see Submitting the batch job below).
• By creating a file-like object and including it in the request (see Specifying the job spec in code below).

When uploading the job spec as a file, the file must be a JSON file that includes:

• The type of information you're requesting (the detail_type). You can specify multiple types. See the list of valid detail types above.
• A list of Reonomy property IDs (the property_ids list).

A sample JSON file requesting basic and ownership information for 10 properties is shown here.

You can optionally include an updated field in the JSON file, indicating that you want only properties with data updated on or after the specified date, in YYYY-MM-DD format (see the example). Any types you specify must also be in the detail_type list. If you specify multiple updated types, the API returns properties where at least one of the types was updated on or after the specified date (OR).

Additionally, you may exclude personally identifiable information (PII) from the results by including "filter_pii": true in the optional filters section of the JSON file, as shown in the example. For more information about PII and PII-related fields, see Excluding personally identifiable information (PII).

Response

It's also possible to create a file-like object with the job specification and upload that with the batch job request. This can be useful if, for example, you used POST /search/summaries to generate a list of properties. The implementation is language specific but the example shows how you might do this in Python.

Response

When a job completes with status SUCCESS, the GET /v2/job/file/{job_id} response includes result_url and result_urls attributes (shown in the example) specifying the URL(s) you can use to download the resulting property data as a file. If you submit more than 25,000 property IDs, the API returns the results in batches of 25,000:

• result_url is a comma-separated string of result URLs.
• result_urls is an array of result URLs.

The file type returned by each result URL is NDJSON. Within the file, each property is represented as a separate JSON object, with a new line (\n) character between each object. Large scale data analysis tools like Spark and BigQuery are optimized for NDJSON and can read files incrementally, which is important since the download size can be very large.

Result URLs are valid only for 10 minutes after the job completes, so be sure to download the results as soon as you see 'status': 'SUCCESS'. If you don't, you'll get an error page indicating that The specified key does not exist. If this happens, call GET /v2/jobs/file/{job_id} again to get a new URL.

As noted above, each result URL returns an NDJSON file. Within a file, each property is represented as a separate JSON object, and a new line (\n) character separates each object.

The example on the right shows an NDJSON file with two properties that's been converted to an array of JSON objects. If you expand one of the top level nodes, you can see the structure of the JSON object for the property. In this case, we requested all data types (basic, mortgages, ownership, etc.) in the job spec file, so all the available Reonomy property data is included.

Within the property object, the order of items is arbitrary. If you scroll through the object, you'll see a node for each requested data type, except that basic attributes (active, municipality, etc.) are displayed at the top level, not under a basic node. Additionally, the ownership contact information is listed under its own contacts node.

Although an NDJSON file includes valid JSON objects, the file itself isn't valid JSON, so you can't read it using a standard JSON reader. However, for relatively small files, you can read the file line-by-line and process each line using a standard JSON reader. The example shows how you can convert an NDJSON file to a standard JSON list using Python.

If you receive an HTTP 400 FileNotFoundAtPathError, it's likely your POST request is not formatted as required by the API.

To determine what's wrong with the request, you might find it helpful to send the POST request to https://httpbin.org/post to see how the client library you're using is constructing the HTTP request. httpbin.org is a service that returns the request to you so you can see exactly how it was formatted by your client library. This can help identify requests that are improperly formatted or libraries that are incompatible. An example of a good response from https://httpbin.org/post is shown here.

Using Node.js

Some Node.js HTTP libraries (for example, node-fetch and axios) do not natively support multipart/form-data streams and instead recommend using the form-data package to submit forms and file uploads. However, form-data creates objects that are incompatible with the Reonomy API and you may receive an HTTP 400 FileNotFoundAtPathError error. Additionally, see https://github.com/axios/axios/issues/789 for information about axios incompatibilities.

Node.js request, request-promise, and request-promise-native all support multipart/form-data streams and work with the Reonomy API. However, they do not by default include a User-Agent header, which is required by the Reonomy API, so be sure to set this manually, as shown in the examples.

For addresses, the CSV file you upload must have at a minimum four columns:

• One for the street address
• One for the city name
• One for the state abbreviation
• One for the zip code

For location coordinates, the CSV file must have at least the following columns:

• One for the latitude value
• One for the longitude value

You can create the CSV file using a standard text editor as shown, but more typically you'll export data as a CSV file from a database or an application like Excel. Make sure you include a header row with the column names.

The CSV file can have as many additional columns as you want. However, if you have additional columns in your CSV file, you must include them all in the headers list when you construct the API request (see below).

Mapping the columns

When you upload a match job, you must tell the API how your CSV file column headers map to the required API parameter names. The first table shows the four required address parameters and how they map to the column names used in the address examples above. The second table shows the required location parameters and how they map to the column names used in the location example above.

You define the mappings in a metadata object you submit with your POST request. The metadata object must include the following attributes:

• headers specifies the names of all the columns in your CSV file, in the correct order.
• mapping specifies the required parameters and the columns they map to in your CSV file.

See the sample code below for an example of the required format.

Response

Response

When a job completes with status SUCCESS, the GET /v2/job/file/{job_id} response includes a result_url attribute (shown in the example) specifying the URL you can use to download the results as a CSV file.

The results file includes everything in the original CSV file, plus the following columns:

The example below shows the resulting CSV file for an upload CSV file that included additional columns.

Match job output files are delivered to a location you can access via SFTP using your API access key and secret.

You can use any SFTP client to access the files:

1. Connect to sftp://sftp.reonomy.com on port 22 using your API access key as the username and your secret key as the password.
2. On the remote site, open the jobs/match folder. You'll see your match job output files listed.
3. Download the file you want. Files are named using the batch job ID.

If you receive an HTTP 400 FileNotFoundAtPathError, it's likely your POST request is not formatted as required by the API. To determine what's wrong with the request, you might find it helpful to send the POST request to https://httpbin.org/post to see how the client library you're using is constructing the HTTP request.

Looking at the response from httpbin.org can help identify requests that are improperly formatted or libraries that are incompatible. An example of a good response from httpbin.org is shown here.

Using Node.js

See Using Node.js in the previous section for information about using Node.js HTTP libraries.

Requests to POST /search/summaries return a maximum of 10,000 property IDs in the initial response. If the search yields more properties than the limit, the API returns a search_token you can use to request additional matching items. This requires re-running the request, possibly multiple times, with the search_token as a query parameter (see Pagination).

As an alternative, you can run a bulk search job, (a batch job with "job_type": "search"), where you upload a JSON file with the search criteria (see below).

As with the other batch job types, the search job runs asynchronously, so you must poll the GET /jobs/file/{job_id} endpoint to determine when it is finished. When the job completes, GET /jobs/file/{job_id} returns a results_url you can use to download the results.

The search file you upload with the POST request is a JSON file that specifies the search criteria, as shown in the example. The format is exactly the same as that used by the POST /v2/search/summaries request body.

One way to create the required JSON is to use the POST /search/summaries page to build your search interactively, then copy the request body into a file and save it with a .json extension.

Response

When a job completes with status SUCCESS, the GET /v2/job/file/{job_id} response includes result_url and result_urls attributes (shown in the example) specifying the URL(s) you can use to download the resulting property data as a file:

• result_url is a string with the result URL.
• result_urls is a single element array with the result URL.

Result URLs are valid only for 10 minutes after the job completes, so be sure to download the results as soon as you see 'status': 'SUCCESS'. If you don't, you'll get an error page indicating that The specified key does not exist. If this happens, call GET /v2/jobs/file/{job_id} again to get a new URL.

The file type returned by the result URL is NDJSON. Within the file, each property is represented as a separate JSON object, with a new line (\n) character between each object.

The example on the right shows a JSON object representing a single property from the results. It provides all of the data update times for the property, as well as the property ID (id).

Although an NDJSON file includes valid JSON objects, the file itself isn't valid JSON, so you can't read it using a standard JSON reader. However, if the file is not too large, you can read it line-by-line and process each line using a standard JSON reader. The example shows how you can convert an NDJSON file to a standard JSON list using Python.

Response