Learning how to learn
I started off asking Claude to help me become a cracked engineer like Elon musk where i learn to build large complex systems varying from rockets to system architectures
Starting off with a System Design question: Design a SD for handling realtime telemtary data from iot devices.
So i came up with a very vague and simple solution which goes like (IOT)---> Server---> {user} this is how the data might be transmitted.
Questions:
Handling data
- Cron jobs are typically used for scheduled tasks on servers, not for IoT devices.
- For real-time data, we often use protocols designed for IoT, like MQTT (which you correctly identified in your diagram).
- Consider: What if data needs to be sent more frequently than every 10-15 minutes? How would this affect your system?
- We can send the data continuously in a stream but what about storage? Do we really need the continuous stream of data all the time?
- What about the latency, bandwidth and power consumption?
- What about security coz the data keeps flowing through the network?
Scalability:
- Database choice is indeed crucial for scalability.
- Load testing ("testing on multiple payloads") is a good practice.
- Modular design and avoiding bottlenecks are key concepts.
- Caching can indeed help with scalability.
Points to work:
- What specific database technologies might be suitable for this use case?
- TimeSeries databases such as influxdb or timescaledb might be the efficient here, coz we'll be recording a lot of telemetry data using timestamps.
- How might you implement caching in this system?
- Consider the concept of horizontal vs. vertical scaling. How might each apply here?