Included with your API key is the ability to download the complete animal and organization database via FTP.  This is the easiest way to do a bulk download of the adoptable pets and rescue/shelter (org) information.

Setup

To enable your FTP account, use the link in the API Key Update email that you will receive on a monthly basis.  You can enable the FTP data download.  If you do not want the files to be compressed please let us know so we can make that change to your API export.

Data files

The nightly export data set includes the following files.  Each data file contains a maximum of 10,000 rows.

• All organization data: orgs_n.json
• All animal data: pets_n.json
• Updated pets from the past 24 hours: updatedpets_n.json
• New pets from the past 24 hours: newpets_n.json
• A list of all available animalIDs: petlist.csv

Data file format

The data files are formatted in JSON format, with the exception of the file petlist.csv (which is a CSV).  Currently the data files can either be compressed or not (zip format), and you can make that selection on your API Key setup page.

Each of the JSON files have a maximum of 1,000 records.  There is one record per row in the data files.

The petlist.csv file has 100 comma separated animalIDs per row.

Please refer to the sample data files that are attached to this article for specific examples.

Schedule

The data files are generated and placed in your dedicated FTP account by 1 AM eastern.  As our data set grows, however, the time that the data files will be available is expected to move later, so you may want to schedule your download and processing at a later time (like 3 or 4 AM eastern).

Your import process

Following is an overview of the process you would use to setup your local database and keep it up-to-date on a daily basis:

1. Import the organization and animal data (orgs_n.json and pets_n.json).  This will be a lengthy and resource intensive process.  The animal data is broken up into multiple files to make processing easier.
2. Process all of the pictures from the animal data.  Store the lastUpdated time stamp in your local database from each picture.
3. On a nightly basis, download and process the newpets_n.json and updatedpets_n.json files.
4. On a nightly basis, download and process the petlist.csv file. See Delete/remove unavailable pets below.
5. Compare the mediaLastUpdated timestamp on each animal to determine if existing pets have had picture changes.  See Image/picture related topics below

Delete/remove unavailable pets using the petlist.csv data file

The petlist.csv file contains a list of adoptable/available animalIDs.  There are 100 animalIDs in each row (comma separated).

Each night, you should read the petlist.csv file, and any animalIDs that are not in the file should be removed from your local database.

Pet descriptions

The data files provide two formats for the animal descriptions: animalDescription and animalDescriptionPlain.

animalDescription is the description just the way the user (rescue/shelter) saved it.  It will include HTML (ex., <a>, <div>, <embed>, etc).

The animalDescriptionPlain field contains the animal description with all HTML removed.  It does, however, include line breaks (\n) to preserve paragraph formatting.

Image/picture related topics

You should store the lastUpdated time stamp from the pictures data when you add a pet to your own database.  Then, when processing pets, when you come across a pet that already exists in your database, check the mediaLastUpdated time stamp on the animal to determine if any of the pictures have changed.  If the mediaLastUpdated time stamp is newer than the time stamp that you stored in your local database you should reprocess the pet's pictures.