Bulk URL Scanner API

Bulk Malicious URL Scanner API Documentation

IPQualityScore's bulk malicious URL scanner validation API allows you to upload a large list of URLs or domains for batch processing through our URL scanner API service. This guide will focus on uploading data in a JSON format, but you're free to use any of our available upload methods, such as POST file upload or a remote file URL.

Uploading CSV files to be processed by our system occurs in a series of steps. Each step is briefly discussed below and additional documentation is available on each step on the left menu. The example script on this page can be used as a quick guide for uploading an URL or domain list for processing. Other upload methods are also available. If you have questions on how to implement this system, please feel free to reach out to our support team and we'll be happy to assist.

Step 1: Upload Your List Of URLs To Our CSV API

Start by making a POST request containing URLs (and potentially other data points, see our Malicious URL Scanning API documentation for more information on allowed parameters) to the API URL described below. A few headers are also required to make this request work. Once you've uploaded the request to our servers you should get a response similar to our success example below.

JSON:

Headers
Field	Value
Content-Type	application/json

JSON Body / Data
{ "type": "url", "file_name": "example.csv", "key": "YOUR_API_KEY_HERE", "input": [ [ "https://ipqualityscore.com" ], [ "https://google.com" ] ] }

For example success/failure responses check out our extended uploading a CSV documentation. You should be able to grab the status URL (or ID) from the response and use that for Step 2.

Step 2: Check the status of your upload.

Your CSV will start processing after upload. Depending on the size of your CSV, processing may take a while to complete. You will want to periodically check the status of your CSV's processing by calling our status API. Here's an example call.

JSON:

Step 3: Download your results.

Once your CSV's status returns FINISHED you can then access any download links supplied to retrieve the output file. Results will be in CSV format.

Analyzing the Bulk URL Scanner Results

Please consult the Malicious URL Scanner API documentation for further details on each data point's definition. This page explains best practices for using our URL Scanner API along with a detailed description of the API's retuned data points.

Example Code

<?php

/*
* This example is not intended for production use.
* This script is thread blocking and will wait for your CSV to finish processing.
* We recommend you break this script into multiple parts and execute it via cron.
* Please reach out to our support team if you have any questions.
*/

//Initialise the cURL var
$step_one = curl_init();

//Get the response from cURL
curl_setopt($step_one, CURLOPT_RETURNTRANSFER, 1);

// Set the required headers specifying this is a JSON request.
curl_setopt($step_one, CURLOPT_HTTPHEADER, array("Content-type: application/json"));

// Set the URL to upload a proxy file.
curl_setopt($step_one, CURLOPT_URL, 'https://ipqualityscore.com/api/xml/csv/YOUR_API_KEY_HERE/url/upload');

// Create a POST array that will be converted into JSON.
$postData = [
    'input' => [
        [
            "https://ipqualityscore.com"
        ],
        [
            "https://google.com"
        ]
    ]
];

// Make this a POST request.
curl_setopt($step_one, CURLOPT_POST, true);

// Encode $postData into json
$json = json_encode($postData);

// Send the JSON to the server.
curl_setopt($step_one, CURLOPT_POSTFIELDS, $json);

// Execute the request
$raw_response = curl_exec($step_one);

// Decode the response.
$response = json_decode($raw_response, true);

// Print out the result or error.
if($response['success']){
    echo "CSV uploaded successfully!";

    // Light sanity checking.
    if(!isset($response['status_url'])){
        echo "Error: Unable to retrieve the status URL from the response. Exiting!";
        exit;
    }

    /*
    * At this point we'll retrieve the status in an infinite loop
    * we don't recommend running this on production.
    * https://ipqualityscore.com/documentation/csv-api/check-csv-status
    */
    while(true){
        // Set the URL to retrieve the status page.
        curl_setopt($step_two, CURLOPT_URL, $response['status_url']);

        // Execute the request
        $raw_response = curl_exec($step_two);

        // Decode the response.
        $response = json_decode($raw_response, true);

        // Print out the result or error.
        if($response['success']){
            echo "Current CSV status:".$response['status'];

            // Check if the CSV is finished.

            if($response['status'] === 'FINISHED'){
                echo "CSV is finished processing! Downloading!";

                /*
                * At this point you'll probably want to download the results.
                */

                file_put_contents($response['file_name'], file_get_contents($response['downloads']['all']));
                echo "Downloaded the results!";
                exit;
            } else {
                echo "CSV is not finished. Will sleep for 1 second and try again.";
                sleep(1);
            }

        } else {
            echo "CSV status request failed: ".$response['message'];
        }
    }
} else {
    echo "CSV upload failed: ".$response['message'];
}