- The supplier's stock file is not a small form submission.
- It may contain hundreds of thousands of SKU-and-location quantities, sometimes produced from a wide spreadsheet with one warehouse per column.
- Loading the entire file into memory is risky.
- For a supplier marketplace built on Medusa, we designed the import as a durable data pipeline.
The client problem
The supplier's stock file is not a small form submission.
It may contain hundreds of thousands of SKU-and-location quantities, sometimes produced from a wide spreadsheet with one warehouse per column. While the file is processing, buyers continue adding products to carts and Medusa continues reserving inventory.
Loading the entire file into memory is risky. Updating one row at a time is slow. Treating the import as one global database transaction keeps locks open for too long. Blindly replacing inventory can overwrite reservation state. Emitting an event for every line can overwhelm everything downstream.
For a supplier marketplace built on Medusa, we designed the import as a durable data pipeline. The file is streamed into PostgreSQL staging. SQL validates and consolidates the rows. Seller ownership is resolved before mutation. Inventory levels are upserted in bounded transactions, with replace and add modes that touch stocked quantity but preserve reservations.
The objective is not to make a CSV endpoint bigger. It is to let industrial stock feeds coexist with a live commerce system.
Keep workflow state small and row state durable
The Medusa workflow passes only the import ID, upload-session ID, seller ID and chosen mode between its stages.
It does not carry an ever-growing array of parsed rows through workflow context. The rows live in a dedicated PostgreSQL staging table keyed by import and source row.
This separation is fundamental. Workflow state describes the operation. Staging state describes the data being processed. A worker can finish one stage and another can continue from durable rows without serialising the entire supplier feed into an orchestration payload.
The pipeline has five responsibilities: validate the request, acquire the seller import lock, stage the stream, validate the staged set and process valid rows.
Each stage can report progress and failure in language meaningful to the operator rather than exposing one opaque request timeout.
Stream the upload instead of buffering it
The staging step opens the stored upload as a readable stream and pipes it through CSV parsing.
Rows are normalised into a canonical long shape: SKU, stock location and quantity. Suppliers with wide spreadsheets can use a saved mapping that expands location columns into those canonical rows before they enter staging.
The importer buffers a bounded group of staged rows. When the group reaches 1,000, it pauses the parser, inserts the batch with one set-based PostgreSQL statement and resumes the stream after the flush succeeds.
The file itself is never assembled as one application array. Memory follows the staging batch rather than total file size.
Required headers are checked early. Effectively blank rows are ignored. Missing, non-numeric or negative quantities are retained as skipped outcomes so the final report can explain why they did not affect stock.
Use staging as the reviewable boundary
Writing parsed rows directly into inventory_level would mix file interpretation with commerce mutation.
Staging gives the platform a boundary where it can inspect, normalise and reject data before live inventory changes. Every useful row retains its source position, raw canonical data, SKU, location, quantity, status and error.
That supports both scale and accountability. SQL can operate over the complete import set without keeping it in Node.js memory. Operations can receive counts for processed, skipped and failed rows. A failed row remains explainable after successful rows have moved on.
The staging table is not a second inventory source of truth. It is the operation ledger for moving one supplier feed into Medusa Inventory.
Define duplicate semantics before deleting duplicates
Supplier files can repeat the same SKU and location for different reasons.
A repeated source key represents the same input contribution more than once, so later copies are skipped. Distinct source keys may legitimately contribute separate quantities to the same SKU/location pair, especially after a wide file is expanded. Those quantities are summed into the earliest canonical row, and later rows are marked as aggregated.
This produces one mutation candidate per inventory level without silently discarding legitimate contributions.
The rule also makes replace and add modes understandable. Replace sets the inventory level to the import's consolidated quantity. Add increments the existing stocked quantity by that consolidated amount.
Without an explicit duplicate contract, replace mode can depend on row ordering and add mode can double-count a repeated file line.
Validate seller ownership in SQL
An inventory SKU is not globally trustworthy merely because its text matches.
The validation step checks that the inventory item is linked to the seller running the import. It prefers an exact case-insensitive SKU match and can fall back to a known four-character supplier suffix convention. Soft-deleted inventory items and seller links are excluded.
The stock location must also belong to that seller. A valid location ID from another tenant is still invalid for this import.
These checks run as set-based SQL updates over staged rows. Invalid quantity, unknown SKU and unowned location become explicit skipped reasons.
The same seller-scoped SKU resolver is used again by the inventory upsert. Validation and mutation therefore do not choose different inventory items when duplicate or suffixed candidates exist.
Tenant ownership is enforced where the data changes, not trusted from a spreadsheet or browser mapping.
Allow only one active import per supplier
Two concurrent replace imports for one supplier have no sensible last-writer story. Two add imports can be valid in theory but are difficult to reconcile with duplicate uploads and operator expectations.
The project serialises inventory imports per seller.
Inside one PostgreSQL transaction, it acquires an advisory lock derived from the seller identity, checks whether another import record is already processing for that seller and moves the current record into processing. A short lock timeout turns contention into a conflict response rather than leaving a request waiting indefinitely.
The lock protects the check-and-transition decision. It does not remain held for the entire multi-stage import. The durable processing state prevents a second import from starting after the acquisition transaction completes.
Different suppliers can still process independently. The boundary matches the business ownership of the stock feed.
Upsert inventory in bounded transactions
After validation, the processor repeatedly selects the next 500 pending staging rows.
For each chunk, one transaction resolves the seller-owned inventory items, upserts their inventory levels and marks those exact staging rows processed. If the transaction rolls back, the rows do not falsely appear completed. This is particularly important for add mode: a successfully applied chunk must not be added again because its staging status failed to advance.
Replace mode assigns the imported stocked quantity. Add mode calculates the existing stocked quantity plus the imported quantity.
Both modes deliberately leave reserved and incoming quantities alone. Reservations represent live commerce activity; they are not supplier feed fields. An import changes what is stocked, not what current carts have already reserved.
The upsert orders inventory item and location keys consistently to reduce deadlock risk when checkout and import activity touch overlapping inventory records.
Be honest about the transaction boundary
The complete file is not one atomic transaction.
Each processing chunk is atomic. Earlier successful chunks remain applied if a later chunk fails or the operator cancels the run. Failed chunks are rolled back, then their staging rows are marked failed outside the dead transaction.
That trade-off is deliberate. A million-row transaction would hold locks and database resources for an impractical period. Chunking gives bounded lock duration, visible progress and local failure reporting.
The operator needs to understand the outcome as a batch operation with row and chunk results, not a binary promise that every line changed simultaneously.
When some chunks succeed and others fail, the import can complete with recorded failures. When every processing chunk fails, the run is failed. Cancellation receives its own terminal state.
Let the operator cancel between chunks
Before selecting each new chunk, the processor re-reads the import record.
If the operator has cancelled it, remaining pending rows are marked skipped with a cancellation reason. No new chunk begins. Successfully committed chunks remain, and the final status reports what was processed before cancellation.
This is a practical kill switch rather than an impossible promise to interrupt a transaction halfway through its SQL statement.
Progress is written by phase: staging, validation, inventory update and terminal outcome. Processed, failed, skipped and total counts give the supplier team a concrete result to investigate.
Long-running work becomes an operation with state, not a browser request that somebody is afraid to close.
Emit one bulk signal, not a million row events
Downstream search and marketplace stock visibility need to know that inventory changed.
They do not need the importer to publish one event for every CSV row.
After a non-cancelled run processes at least one row, the workflow emits one small event containing the seller, import and processed count. Downstream code can use the import identifier to resolve affected products and reconcile its own projection.
A separate terminal event covers completed, failed and cancelled outcomes so email-ingestion or notification flows can finish their own lifecycle.
This is a bulk-work contract: point consumers at the durable operation rather than copying the entire changed dataset into an event bus.
It avoids an event storm without claiming universal exactly-once delivery.
Why Medusa was the right inventory foundation
Medusa already models inventory items, locations, stocked quantity, reservations and incoming stock. The marketplace needed an industrial ingestion path around those concepts, not a parallel stock database.
The custom workflow adds supplier ownership, file mappings, durable staging, validation, progress and bounded mutation. The final writes still respect Medusa's inventory-level model.
That is the architectural advantage. We could adapt the ingestion system to the client's supplier feeds while preserving the commerce semantics checkout depends on.
An extensible engine lets the platform accept messy real-world operations without making the core inventory model messy too.
A high-volume inventory-import checklist
Before accepting a large supplier feed, ask:
- Is the upload streamed or fully buffered?
- Where do rows live between workflow stages?
- Can wide supplier files map to one canonical shape?
- What does a duplicate source row mean?
- Are quantities aggregated per SKU and location?
- Is SKU resolution scoped to the seller?
- Is the stock location owned by that seller?
- Does the operator choose replace or add semantics?
- Are reserved and incoming quantities preserved?
- Can two imports run for the same supplier?
- What is the transaction boundary?
- Are mutation and staging completion in the same chunk transaction?
- Can cancellation stop before the next chunk?
- Are error details and progress bounded and visible?
- Does downstream processing consume one bulk pointer?
If those answers are absent, file size is only the first problem the import will expose.
The broader lesson
Scaling inventory imports is not mainly about accepting a larger upload limit.
It is about separating ingestion from mutation, making ownership explicit, defining duplicate and update semantics, preserving live reservations and choosing a transaction boundary the database can sustain.
For this marketplace, we built a streamed staging pipeline and set-based validation before bounded Medusa Inventory upserts. The system can process very large feeds with memory tied to batch size, while operators see progress and the commerce model remains authoritative.
That is how a supplier file becomes a controlled marketplace operation instead of a risky database script.