Programmatically Download Session Stream Files

Session Stream provides open access to raw Monetate session data. You can use this data in relational databases such as MySQL, data warehouses such as Amazon Redshift, or BI platforms such as Tableau and Domo.

Session Stream provides JSON data files compressed with Gzip in 15 minute increments. Each file contains raw data from every session that Monetate observed on your site during that time. Session Stream is available for all Monetate users.

For more information about the fields each file contains, how to obtain the files via SFTP, and more, see the Session Stream documentation in the Monetate Knowledge Base.

As an alternative to using an SFTP client, you can use the following example Python script to download your Session Stream files.

You must have Paramiko 2.1.1 or later to run this example script.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from argparse import ArgumentParser
from datetime import datetime, timedelta
import paramiko
import os

PATH_FORMAT = "%Y/%m/%d"


def main():
    yesterday = datetime.now() - timedelta(days=1)
    parser = ArgumentParser(description="Downloads monetate session stream via FTP for the specified date.")
    parser.add_argument("--host", default="download.monetate.net", help="The host to connect to (default: %(default)s)")
    parser.add_argument("--port", default=22, type=int, help="The port to connect on (default: %(default)s)")
    parser.add_argument("-u", "--username", required=True, help="FTP user to connect as")
    parser.add_argument("-p", "--password", required=True, help="FTP user password")
    parser.add_argument("-d", "--date", default=yesterday.strftime(PATH_FORMAT),
                        help="The date to sync locally, format YYYY/MM/DD (default: %(default)s)")
  parser.add_argument("--root_output_dir", default=os.path.abspath(os.path.dirname(__file__)),
                        help="The root directory where files will be synced locally (default: %(default)s)")
    args = parser.parse_args()

    base_path = "raw_data/session_stream/{}".format(args.date)

    # create local paths if they don't exist
    try:
        output_path = os.path.join(args.root_output_dir, base_path)
        os.makedirs(output_path)
    except OSError:
        pass

    # list the files in the currently directory
    local_files = frozenset(os.listdir(output_path))

    # connect via ftp and list the files
    transport = paramiko.Transport((args.host, args.port))
    transport.connect(username=args.username, password=args.password)
    sftp = paramiko.SFTPClient.from_transport(transport)

    # list the files we need to download
    print "Calculating files to download..."
    current_files = frozenset(sftp.listdir(base_path))
    files_to_download = current_files - local_files

    # download all the files needed
    print "Fetching {} files...".format(len(files_to_download))
    for filename in files_to_download:
        remote_path = "{}/{}".format(base_path, filename)
        local_path = os.path.join(output_path, filename)
        sftp.get(remote_path, local_path)

if __name__ == '__main__':
  main()

When run more than once for a particular date, this script only downloads files that you haven't already retrieved. You can input as a parameter the date for which you want to fetch data. The Session Stream files are downloaded to the location you specified with the same date-based directory structure that Monetate uses to organize the files. You can schedule this script to run daily or write a similar application that better meets your needs.