I’ve been messing around with Low latency and high frequency message processing and figured it might be worth sharing. I created an example of the High-Speed Subscribers (Black Box Pattern) using ZeroMQ, “a high-performance asynchronous messaging library”. According to its creator, you can pipe through 2 mil to 6 mil+ messages per second with one CPU core.
For some perspective, the New York Stock Exchange generates approx. 4 terabytes of data and in excess of 15 billion messages over the course of a single 6.5-hour trading day. If you think that’s a lot, in Oct 2008 the NASDAQ TotalView -ITCH produced 999,814,794 messages in a single day.
High-Speed Subscriber Patterns in ZeroMQ
- ZeroMQ Guide: http://zguide.zeromq.org/py:chapter5#toc5
Setup Environment & Run Example (Windows):
git clone https://github.com/ryansmccoy/zeromq-high-speed-subscribers
cd zeromq-high-speed-subscribers
conda create -n zeromq-high-speed-subscribers1 python=3.7 -y
activate zeromq-high-speed-subscribers1
pip install -r requirements.txt
Setup Environment & Run Example (Linux):
git clone https://github.com/ryansmccoy/zeromq-high-speed-subscribers
cd zeromq-high-speed-subscribers
conda create -n zeromq-high-speed-subscribers python=3.7 -y
pip install -r requirements.txt
To run, open seperate terminal windows, run:
Publisher
$ activate zeromq-high-speed-subscribers $ python zeromq_high_speed_subscribers/00_publish_data_feed.py Output:
Subscriber-to-Pusher
$ activate zeromq-high-speed-subscribers $ python zeromq_high_speed_subscribers/01_subscriber_to_pusher_queue.py Output:
Pull-to-Workers
$ activate zeromq-high-speed-subscribers $ python zeromq_high_speed_subscribers/02_pull_to_workers.py Output:
