Programmatically Download Session Stream Files

session stream provides open access to raw monetate session data you can use this data in relational databases such as mysql, data warehouses such as amazon redshift, or bi platforms such as tableau and domo session stream provides json data files compressed with gzip in 15 minute increments each file contains raw data from every session that monetate observed on your site during that time session stream is available for all monetate users for more information about the fields each file contains , how to obtain the files via sftp , and more, see the session stream documentation in the monetate knowledge base as an alternative to using an sftp client, you can use the following example python script to download your session stream files you must have paramiko 2 1 1 or later to run this example script example script to download session stream files #!/usr/bin/env python \# coding utf 8 from argparse import argumentparser from datetime import datetime, timedelta import paramiko import os path format = "%y/%m/%d" def main() yesterday = datetime now() timedelta(days=1) parser = argumentparser(description="downloads monetate session stream via ftp for the specified date ") parser add argument(" host", default="download monetate net", help="the host to connect to (default %(default)s)") parser add argument(" port", default=22, type=int, help="the port to connect on (default %(default)s)") parser add argument(" u", " username", required=true, help="ftp user to connect as") parser add argument(" p", " password", required=true, help="ftp user password") parser add argument(" d", " date", default=yesterday strftime(path format), help="the date to sync locally, format yyyy/mm/dd (default %(default)s)") parser add argument(" root output dir", default=os path abspath(os path dirname( file )), help="the root directory where files will be synced locally (default %(default)s)") args = parser parse args() base path = "raw data/session stream/{}" format(args date) \# create local paths if they don't exist try output path = os path join(args root output dir, base path) os makedirs(output path) except oserror pass \# list the files in the currently directory local files = frozenset(os listdir(output path)) \# connect via ftp and list the files transport = paramiko transport((args host, args port)) transport connect(username=args username, password=args password) sftp = paramiko sftpclient from transport(transport) \# list the files we need to download print "calculating files to download " current files = frozenset(sftp listdir(base path)) files to download = current files local files \# download all the files needed print "fetching {} files " format(len(files to download)) for filename in files to download remote path = "{}/{}" format(base path, filename) local path = os path join(output path, filename) sftp get(remote path, local path) if name == ' main ' main() when run more than once for a particular date, this script only downloads files that you haven't already retrieved you can input as a parameter the date for which you want to fetch data the session stream files are downloaded to the location you specified with the same date based directory structure that monetate uses to organize the files you can schedule this script to run daily or write a similar application that better meets your needs