eCommerce platform for 20k+ products and frequent updates
Hello fellow warriors
Being a long term reader of this forum, it's now time for my first post here!
I'm in the setup-phase of an ecommerce store with a dropship-like business model, and need help finding a matching ecommerce platform.
My supplier doesn't have the regular dropship-supplier support with an API onto which I can connect my ecommerce system, but instead offers .txt-files on an FTP-site with their catalouge, price and stock quantity in separate files. On this FTP-site they also accept orders to be in .txt-files with a defined structure on the information. They then process these files every 4 hours or so and send my customers their order accordingly.
So far I've made a setup with WooCommerce (this is the platform I'm most familiar with and run other stores on today) running on a AWS machine with a plugin called WP All Import in order to do the import of the catalogue and prices every saturday night, and the stock quantity every 2 hours. The files are locally stored on the machine but updated as required from a CRON-script running locally. The plugin then sorts the import process out and does this according to a schedule.
BUT, the big problem here is that the import process takes a LOT of time, importing the catalogue from scratch takes a good 4-5 hours, updating it every weekend takes 3-4 hours + 2-3 hours for prices and then the stock quantity takes about 1h to finish (remember, run every 2 hours which makes the stock questionable...). In the wake of this comes problems like the site becoming unresponsive due to the seemingly heavy process of updating all this, products missing prices and/or stock quantity or products not being deleted as they should when not available in the catalogue anymore.
Talking to friends who work with databases hauling large amounts of data, they believe this is a ridicously long time for and import of such small amounts of data, and points out that the machine running this is well within performance and that the fault must be within the system (i.e wordpress and/or WooCommerce+plugin).
So, long story short, has anyone here experience in doing sites with this amount of products and the updating process of them? Thoughts on my current setup?
What other ecommerce platform would you recommend I look into that would perform much better?
I'm open and thankful for all your thoughts on this!
Being a long term reader of this forum, it's now time for my first post here!
I'm in the setup-phase of an ecommerce store with a dropship-like business model, and need help finding a matching ecommerce platform.
My supplier doesn't have the regular dropship-supplier support with an API onto which I can connect my ecommerce system, but instead offers .txt-files on an FTP-site with their catalouge, price and stock quantity in separate files. On this FTP-site they also accept orders to be in .txt-files with a defined structure on the information. They then process these files every 4 hours or so and send my customers their order accordingly.
So far I've made a setup with WooCommerce (this is the platform I'm most familiar with and run other stores on today) running on a AWS machine with a plugin called WP All Import in order to do the import of the catalogue and prices every saturday night, and the stock quantity every 2 hours. The files are locally stored on the machine but updated as required from a CRON-script running locally. The plugin then sorts the import process out and does this according to a schedule.
BUT, the big problem here is that the import process takes a LOT of time, importing the catalogue from scratch takes a good 4-5 hours, updating it every weekend takes 3-4 hours + 2-3 hours for prices and then the stock quantity takes about 1h to finish (remember, run every 2 hours which makes the stock questionable...). In the wake of this comes problems like the site becoming unresponsive due to the seemingly heavy process of updating all this, products missing prices and/or stock quantity or products not being deleted as they should when not available in the catalogue anymore.
Talking to friends who work with databases hauling large amounts of data, they believe this is a ridicously long time for and import of such small amounts of data, and points out that the machine running this is well within performance and that the fault must be within the system (i.e wordpress and/or WooCommerce+plugin).
So, long story short, has anyone here experience in doing sites with this amount of products and the updating process of them? Thoughts on my current setup?
What other ecommerce platform would you recommend I look into that would perform much better?
I'm open and thankful for all your thoughts on this!
