A reusable framework for large CSV imports and exports
I designed and implemented a reusable framework for large CSV imports and exports. It uses streaming, partitioned processing and persistent checkpoints to limit memory use and support recovery after interrupted processing. It handles hundreds of thousands of records from CSV files and databases on workers limited to 500 MB of memory.
Swipe to see the full diagram
- Volume
- hundreds of thousands of records
- Worker memory
- 500 MB maximum
- Sources
- CSV files and databases
- Uses
- streaming, partitions, checkpoints
- Recovers
- after interrupted processing