Build your own bot to harvest market data

How can I get A LOT MORE cryptocurrency historical data for my machine learning modelling?

Cryptodata — a data harvesting bot

“two green and red cultivators on brown field” by Heiko Janowski on Unsplash

What it does

  1. The cryptodata bot wakes up every period (currently configured to be every 1 minute).
  2. It then collects the current market data from a number of exchanges at the same time using their REST APIs.
  3. The data collected is then saved to a MongoDB database in a fairly consistent structure.
  4. The bot then goes back to sleep until it wakes up again at the start of the next minute.

What data do we collect?

  • latest market prices
  • current open order books
  • recent trade history

Structuring the MongoDB database

from mongoengine import Document, connect, FloatField, StringField, DictField, ListFieldfrom config import DATABASEconnect(DATABASE, host='localhost', port=27017)
class Ticker(Document):
"""
Document to store ticker information
"""
unixtime = FloatField(required=True, )
exchange = StringField(required=True, max_length=20)
pair = StringField(required=True, max_length=10)
content = DictField(required=True)
class Orderbook(Document):
"""
Document to store orderbook information
"""
unixtime = FloatField(required=True, )
exchange = StringField(required=True, max_length=20)
pair = StringField(required=True, max_length=10)
content = DictField(required=True)
class Trades(Document):
"""
Document to store recent trades information
"""
unixtime = FloatField(required=True, )
exchange = StringField(required=True, max_length=20)
pair = StringField(required=True, max_length=10)
content = ListField(required=True)

Some more technical details

Hardware

Installation and packages

  1. MongoDB installed and with the Mongo server running
  2. A Python 3.6 virtual environment for the bot with the packages you need (details are in the requirements.txt file)

Collecting the data

Parallelisation

How much data might we get?

GitHub repository

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matt Gosden

Matt Gosden

Insurance meets tech meets music. #fintech #AI