Nice write-up, explored a lot of different technology.

Would like to drop a note as I wonder if the whole steps could be simplified with real-time api to write data and analyze data.

Check out this to see if it's interesting - https://columns.ai/build

Note that, I'm not selling things, just as the same passionate as you in real time big data analytics, so feel free to reach out if you want to discuss.