Versions Compared

Old Version 5

changes.mady.by.user Richard [RG]

Saved on

compared with

New Version Current

changes.mady.by.user Richard [RG]

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Included with your API key is the ability to download the complete animal and organization database via FTP.  This is the easiest way to do a bulk download of the adoptable pets and rescue/shelter (org) information.

Setup

To enable your FTP account, use the link in the API Key Update email that you will receive on a monthly basis.  You can enable the FTP data download.  If you do not want the files to be compressed please let us know so we can make that change to your API export.

Info

Please contact RescueGroups.org support if you have not received a link to your API Key configuration.  The email is sent out on a monthly basis.

Data files

The nightly export data set includes the following files.  Each data file contains a maximum of 10,000 rows.

  • All organization data: orgs_n.json
  • All animal data: pets_n.json
  • Updated pets from the past 24 hours: updatedpets_n.json
  • New pets from the past 24 hours: newpets_n.json
  • A list of all available animalIDs: petlist.csv
  1. Request an API key from RescueGroups.org.
  2. Enable your FTP data export.
  3. Download your data files.
  4. Perform the initial import of the data.
  5. Perform daily updates of the data.
  6. Perform regular scrubs of the data.

Request an API key from RescueGroups.org

You can request an API Key here:

https://www.rescuegroups.org/services/request-an-api-key/

Enable your FTP data export

Soon after your API Key is created you will receive an email with a link to update your API key information.  The API key edit page will also include the option to enable the FTP-based download of the pet data.

  1. Click the link you received in the email.
  2. Next to Enable daily JSON data file download via FTP select Yes.
  3. Click Save.

Your FTP credentials will be emailed to you.  You can use that FTP account to download the full adoptable pet data.

Info

Your API key's data files will be available by 6 AM Eastern the day after you enable the FTP-based export.

Download your data files

Login to the FTP account you created above and download the following files:

  • orgs_n.json - all organization data
  • pets_n.json - all animal data
  • updatedpets_n.json - updated pets from the past 24 hours
  • newpets_n.json - new pets from the past 24 hours
  • petlist.csv - a list of all available animalIDs
Info

Multiple files may be generated for the organization, animal, new pets, and updated pets files. The file names will be numbered (e.g., orgs_1.json, orgs_2.json).

Perform the initial import of the data

Import the data from orgs_n.json and pets_n.json into your database.

Info

Please see the section Image/picture related topics below for important information on how to handle pictures and picture changes.

Perform daily updates of the data

On a daily basis, download the adoptable pet data from your API key's FTP account.  Process the updated data.

  1. Process the data for each pet in the updatedpets_n.json files.
  2. Add the pets that are in the newpets_n.json files.
  3. Delete any pets (by animalID) that are not listed in the petlist.csv file.

Perform regular scrubs of the data

On a regular basis (e.g., weekly or monthly), perform a full refresh of data from the orgs_n.json and pets_n.json data files.

Additional information about the data files

Each data file contains a maximum of 10,000 rows.

Info

The orgs_1.json file (the first file in the orgs data set) contains a time stamp as the first row.  The time stamp is the date/time that the export was started.

...

Please refer to the sample data files that are attached to this article for specific examples.

Schedule

The data files are generated and placed in your dedicated FTP account by 3 AM eastern.  As our data set grows, however, the time that the data files will be available is expected to move later, so you may want to schedule your download and processing at a later time (like 4 or 5 AM eastern).

Your import process

Following is an overview of the process you would use to setup your local database and keep it up-to-date on a daily basis:

  1. Import the organization and animal data (orgs_n.json and pets_n.json).  This will be a lengthy and resource intensive process.  The animal data is broken up into multiple files to make processing easier.
  2. Process all of the pictures from the animal data.  Store the lastUpdated time stamp in your local database from each picture.
  3. On a nightly basis, download and process the newpets_n.json and updatedpets_n.json files.
  4. On a nightly basis, download and process the petlist.csv file. See Delete/remove unavailable pets below.
  5. Compare the mediaLastUpdated timestamp on each animal to determine if existing pets have had picture changes.  See Image/picture related topics below

Delete/remove unavailable pets using the petlist.csv data file

The petlist.csv file contains a list of adoptable/available animalIDs.  There are 100 animalIDs in each row (comma separated).

Each night, you should read the petlist.csv file, and any animalIDs that are not in the file should be removed from your local database.

Pet descriptions

The data files provide two formats for the animal descriptions: animalDescription and animalDescriptionPlain.

...