TL;DR: Here's how I discovered which data-center simultaneously hosts groups of cryptocurrency exchanges. The objective was to identify the best facilities to host a high-frequency trading bot.

I had a conversation with a friend about a mandatory trading firm's competitive advantage: having low latency access to exchange markets. Trading companies achieve low latency by leveraging ad-hoc physical infrastructure that allows them to access exchange markets fast, usually by investing upfront on their infrastructure or paying someone else to use theirs. Unfortunately, those are entry barriers for someone like me playing around with technology. So I wondered how that was in the world of cryptocurrencies.

Cryptocurrency exchanges are privately owned web platforms that allow users to exchange different cryptocurrencies. The first step for a user to use the platform is to transfer cryptocurrency from the user's private wallet to a wallet owned by the exchange. We call those On-Chain transactions since they leverage a blockchain to store all the transactions that happened and the changes in owned amount by each wallet. After the platform verifies that the transaction happened on the blockchain, the user will see the amount increased on his dashboard. From now on, all the cryptocurrency transfers done are off-chain transactions, which means they don't leverage any blockchain at all. They are not based on any blockchain since the exchange holds a private database taking note of the amounts owed to its users. In other words, the exchange database performs as a centralized ledger taking note of its debts toward the users. We call off-chain transactions to distinguish them from the on-chain ones.

This constellation of privately-held centralized ledgers is not the only way to exchange cryptocurrencies: recent developments were using Distributed Finance (DeFi), but maybe I will talk about that in another post.

There are main differences between cryptocurrency exchanges and well-established stock exchanges such as Nasdaq or Euronext:

• Absent regulation: Those places are the heaven for Neoliberalists: the only fees on transactions are the only ones enforced by platform owners.

• Security threats: It's frequent to see those platforms as victims of cybercriminals attacks. Cybercriminals aim to steal the crypto assets owned by the exchanges. That translates into higher risks for firms trading crypto assets than using classical finance.

• Higher volatility: Compared with established classical stocks, cryptocurrencies have a bigger chance to change their value. Any crypto trading firm should face the reality that all the assets they invested in can suddenly change in value. To mitigate this phenomenon, a cryptocurrency finance firm introduced a coin designed to have the same value of US dollars: USD Coin.

One type of financial activity is Risk-Free arbitrage, in its simplest form is the strategy of buying an asset and resell it at a higher price. Trading firms leverage ad-hoc infrastructure to have low latency toward exchange markets, also known as High-Frequency Trading.

Another type of financial activity trading firms perform is Market Making. The idea is to create buy and sell orders of the same asset simultaneously but with a price difference, also called spread; the presence of small independent actors consuming orders will lead the firm to perform a revenue. An interesting open-source project already implementing this strategy is Hummingbot. Unfortunately, the project provides only a few connectors toward a small group of over-competitive exchanges, such as Binance or Bitfinex. On the other hand, it offers many strategies such as Cross Exchange Market Making: a mix of Market Making and Arbitrage.

The post's idea is to find groups of exchanges located in the same datacenter. A trading bot using Cross Exchange Market Making strategy can make money by running in the same datacenter and leveraging low latency toward both exchanges.

I extracted a list of all the API URLs using an open-source library that implements a unified interface toward hundreds of different crypto exchanges: CCXT.

Here's the code I used to extract the URLs from the library:

import pandas as pd
import ccxt
from importlib import import_module

ccxt_modules = [mod for mod in dir(ccxt) if mod[0].islower() and mod not in 
                ['base','decimal_to_precision','error_hierarchy',
                 'errors','exchanges','static_dependencies']]
rows = []
for mod in ccxt_modules:
    full_module_name = "ccxt." + mod

    imp_module = import_module(full_module_name)
    describe = getattr(imp_module, mod)().describe()
    api_elem = describe['urls']['api']
    if isinstance(api_elem, str):
        api_url = api_elem
    else:
        assert isinstance(api_elem, dict)
        if 'private' in api_elem.keys():
            api_url = api_elem['private']
        elif 'v3Private' in api_elem.keys():
            api_url = api_elem['v3Private']
        elif 'trade' in api_elem.keys():
            api_url = api_elem['trade']
        elif 'api' in api_elem.keys():
            api_url = api_elem['api']
        elif 'public' in api_elem.keys():
            api_url = api_elem['public']
        elif 'rest' in api_elem.keys():
            api_url = api_elem['rest']
        elif 'publicV2' in api_elem.keys():
            api_url = api_elem['publicV2']
        elif 'current' in api_elem.keys():
            api_url = api_elem['current']
        elif len(api_elem) == 1:
            api_url = api_elem[api_elem.keys()[0]]
        else:
            print("Unexpected json structure")
    if 'hostname' in api_url:
        api_url = api_url.format(hostname=describe['hostname'])

    rows.append([describe['id'], api_url])

exchanges_df = pd.DataFrame(rows, columns=['name','api_url']).set_index('name')
print(f"Got {len(exchanges_df)} URLs:")
exchanges_df.reindex(exchanges_df.api_url.str.len().sort_values().index).iloc[55:65]

Got 124 URLs:

Each exchange market has its API managed by a specific Web Server. I decided to locate each web server by using its IP address:

from urllib.parse import urlparse
import socket

def extract_domain(url):
    domain = urlparse(url).hostname
    return domain

def get_ip(domain):
    try:
        return socket.gethostbyname(domain)
    except Exception as e:
        pass

exchanges_df['api_url_domain'] = exchanges_df.api_url.dropna().apply(extract_domain)
exchanges_df['api_ip'] = exchanges_df['api_url_domain'].dropna().apply(get_ip)
exchanges_df.reindex(exchanges_df.api_url_domain.str.len().sort_values().index)[['api_url_domain', 'api_ip']].head(15)

On the Internet, each IP address belongs to an Autonomous System. Simplifying, an Autonomous System (AS) is a collection of public IP addresses belonging to a single administrative entity. The Internet Assigned Numbers Authority (IANA) assigns a single unique identifier to each Autonomous System, also known as Autonomous System Number (ASN).

I used an open-source library: pyasn, developed by the Economics of Cybersecurity research group at the Delft University of Technology. It allows offline retrieval of an ASN from a generic IP address. It downloads and processes up-to-date routing data (the MRT/RIB BGP archive) and builds local data structures for fast retrieval:

%%capture
!pip install pyasn
!pyasn_util_download.py --latest
!pyasn_util_convert.py --single *.bz2 ipasn.dat
!rm *.bz2

Then, I retrieved each Autonomous System Number from each IP address:

import pyasn

asndb = pyasn.pyasn('./ipasn.dat')

def _get_asn(ip):
    return asndb.lookup(ip)[1] or ip


exchanges_df['api_ip_prefix'] = exchanges_df['api_ip'].dropna().apply(_get_asn)
exchanges_df[['api_ip_prefix']]

Amazon Web Services (AWS) publishes its current IP address ranges in JSON format, containing data regarding which IP prefix corresponds to which AWS region.

import requests

url = 'https://ip-ranges.amazonaws.com/ip-ranges.json'
req = requests.get(url)

aws_ip_ranges_df = pd.DataFrame(req.json()['prefixes'],  
                                columns=['ip_prefix', 
                                         'region', 
                                         'service', 
                                         'network_border_group'])
aws_ip_ranges_df.set_index('ip_prefix', inplace=True)
aws_ip_ranges_df[['region']]

I joined those tables to identify which AWS facility hosted any exchange market.

from IPython.display import display

exchanges_aws_df = exchanges_df.merge(aws_ip_ranges_df, how='left', left_on='api_ip_prefix', right_index=True)

for region, group in exchanges_aws_df[['api_url_domain', 'region']].drop_duplicates().groupby('region'):
    if len(group)>1:
        display(group)

Conclusions

I leave you with this last image that shows two simultaneous pings launched from the same instance in AWS Tokyo: hitting both of them no more than 8 milliseconds.

Thanks for reading and see you at the next post!