Uploading Thousands of Files to Object Storage
How signed URLs send thousands of simultaneous uploads straight to R2, S3, or Google Cloud Storage, past your server.
Picture a product where accounting teams upload scanned receipts and invoices. Most days are quiet. A few people upload a few files each.
The first version of the upload feature is the one most of us would write. There’s a form with a file picker. The browser sends each file to your API, and the API writes it to storage. You test it with twenty files, it works, and you move on.
Then the end of the month arrives.
By nine in the morning, 300 people have each dragged in a folder of about 2,000 files. That’s 600,000 files. At an average of 4 MB, it’s around 2.4 terabytes, and all of it is trying to arrive in the same few hours. These are made-up, round numbers to reason with, not a measurement of any real system.
Here’s what that first version asks your API to do with them.
Every file is a request that stays open for as long as the file takes to arrive. The server reads each body, holds it in memory or streams it along, and writes it out to storage. So every byte crosses your server twice, once on the way in and once on the way out. Meanwhile, the same servers are supposed to be answering everything else your product does. The login page gets slow. So does the dashboard. If the code records progress as it goes, the database starts taking a write for every chunk of every file, too.
Your API has quietly turned into a relay for bytes it never needed to look at.
Where the bytes go
Let’s put numbers on that, for one person’s batch. The figure below starts on the route we just described. Switch to Direct to storage and watch which numbers change, and which one doesn’t.
| Through your API | Direct to storage | |
|---|---|---|
| File bytes through your API | 8 GB in, and again out | none |
| Requests your API serves | 2,000, each open for a whole transfer | 40 short JSON calls, half to sign and half to confirm (2 MB) |
| Database rows to write | 2,000 | 2,000 |
The last row is the interesting one. The database records every file whichever route the bytes take. And remember this is one person. Our month-end morning has 300 of them.
The obvious fixes
The first instinct is to add servers. That helps, in the sense that you can now relay more bytes in parallel. But you’re paying to scale a part of the system whose job, for these requests, is to copy data from one network connection to another.
The second instinct is to raise timeouts, so slow uploads stop failing. That keeps more requests open for longer, which is the opposite of what an overloaded server needs.
Both help a little. Neither gets at why the server is struggling, which is that an upload is really two different jobs tangled together.
Two different jobs
Think about the front desk in an office building. When a courier arrives with boxes for the 14th floor, the front desk does not carry the boxes upstairs. It checks who the courier is, writes the delivery in a log, and hands over a badge that opens one door, for today only. The courier carries the boxes. Later, someone checks that the boxes actually arrived.
Uploads split into the same two jobs.
- The control path is small. It decides who may upload, which files, and where each one goes, and later whether each one arrived intact. Your application has to own this.
- The data path is the file contents themselves. Your application does not have to carry them.
The badge, in our case, is a signed URL. Object storage services such as Cloudflare R2, Amazon S3, and Google Cloud Storage can all issue them. The browser asks your API for permission, sends the bytes straight to storage, and then reports back.
Step through the handoff below. It uses the same map as the figure above. Keep an eye out for the colored line. It only appears once.
The API and database still do real work. They authorize, record metadata, move files between states, and check completions. Each operation is small, and much lighter than relaying the files themselves, but the total still grows with every file. This doesn’t make the upload free, either. The browser still uses its connection, the storage service still receives every byte, and the API still answers all those small requests.
What a signed URL allows
A signed URL grants its holder one specific operation on one specific object until it expires. Anyone who has it can use it, which makes it a bearer secret.
Your server also chooses where each file goes. Generate a random object key under a staging prefix that belongs to the owner. Never build a key from the file’s name, and never let the browser pick a bucket or a path.
Two details matter more than they look. The URL can’t do more than the credentials that signed it, and an upload to a key that already exists replaces that object. AWS’s upload guide (opens in a new tab) states both.
Signing is a widely shared pattern, not a promise that every provider behaves identically. Cloudflare’s R2 presigned URL guide (opens in a new tab) documents which methods R2 supports and how long its URLs may live, and Google’s signed URL guide (opens in a new tab) does the same for Cloud Storage. Read your provider’s page before choosing an expiry. A few minutes is plenty for one upload, as long as each URL is signed close to when its upload starts. A URL that waits behind a long queue can expire before its turn comes.
Sign each request as narrowly as your provider allows. Binding the content type means the browser has to send a matching header. It proves nothing about the bytes, though. The size and type the browser declares are untrusted input, and they get checked again after the upload.
Checking afterwards has a catch. By then, storage has already received the bytes. Someone could ask for a URL for a 1 MB file and send 5 GB instead. Where your provider supports it, have storage enforce a size limit as well. Either way, keep URLs short-lived, and delete anything that fails the check.
Hand out URLs a few at a time
Before returning any URL, the API saves a record for each file. It notes who owns it, which key the server chose, the size and type it expects, and that it’s still pending. Later, these records are how the server knows what it actually authorized. If signing or the response fails after they’re written, expire or clean up the unused ones.
Don’t hand out 2,000 URLs in one response, either. Ask for them a page at a time, as the browser needs them. That keeps each request small for the API and the database, and it limits how many live bearer URLs exist at any moment. Authenticate every request, check that the batch belongs to the caller, rate-limit these requests, and cap both the number of files and their declared sizes.
Now the browser has the problem
We’ve moved the bytes off the server. Congratulations, the browser now has 2,000 files and an Upload button.
The tempting thing is to start all 2,000 uploads at once. It wouldn’t work that way, because the browser has limits of its own (more on those in a moment). But even the attempt moves the bottleneck into the browser, the user’s network, and the storage endpoint. Requests pile up waiting for a turn, the ones that do run split the same connection, and any of them can time out.
The fix is a small pool. Only a fixed number of uploads run at once, and the next file starts the moment a slot frees up.
The replay below follows one synthetic batch through a pool of four, on a flaky connection. Step through it. When a step outlines a tile, that’s the file its sentence is about.
- scan-01.pdf, uploading
- scan-02.pdf, uploading
- scan-03.pdf, uploading
- scan-04.pdf, uploading
- scan-05.pdf, queued
- scan-06.pdf, queued
- scan-07.pdf, queued
- scan-08.pdf, queued
- scan-09.pdf, queued
- scan-10.pdf, queued
- scan-11.pdf, queued
- scan-12.pdf, queued
- scan-13.pdf, queued
- scan-14.pdf, queued
- scan-15.pdf, queued
- scan-16.pdf, queued
- scan-17.pdf, queued
- scan-18.pdf, queued
- scan-19.pdf, queued
- scan-20.pdf, queued
- scan-21.pdf, queued
- scan-22.pdf, queued
- scan-23.pdf, queued
- scan-24.pdf, queued
- Uploading
- 4 of 4
- Uploaded
- 0/24
- Waiting to retry
- 0
- Failed
- 0
The pool never runs more than four uploads, even while files are failing around it. A dropped transfer doesn’t stall the batch, because its slot goes to the next file straight away.
Why four, and not forty?
Honestly? Four is a bet.
In the replay, four is where the model’s connection fills up. Past that point, adding uploads doesn’t make the batch any faster. It just slices the same bandwidth thinner, and every upload hangs around for longer.
| Pool size | Ticks to finish | Speed per upload |
|---|---|---|
| 1 at a time1 MB/tick each | 94 ticks | 1 MB/tick each |
| 2 at a time1 MB/tick each | 48 ticks | 1 MB/tick each |
| 3 at a time1 MB/tick each | 33 ticks | 1 MB/tick each |
| 4 at a time1 MB/tick each | 25 ticks | 1 MB/tick each |
| 6 at a time0.67 MB/tick each | 26 ticks | 0.67 MB/tick each |
| 8 at a time0.5 MB/tick each | 25 ticks | 0.5 MB/tick each |
In a real browser, you often don’t get forty anyway. Over the older HTTP/1.1 protocol, Chrome only opens a handful of connections to the same server at once, around six. Start forty uploads and most of them just sit in a queue inside the browser, waiting for a turn. Newer protocols can squeeze many uploads through a single connection instead, but they all still share the same internet connection.
Those few connections aren’t all yours, either. Thumbnails loading from the same bucket want them. So does the same app open in another tab. How many uploads you can usefully run depends on whatever else is going on at the same time.
That’s probably why upload libraries can’t agree on a number. Their defaults land anywhere from two to six.
So don’t overthink it. Start small, say three or four, and watch how it behaves on real connections. If uploads start timing out, go lower. If the connection sits idle between files, go higher.
What deserves a retry
The replay retried some failures and gave up on one. Those rules matter in practice. Retry network errors, timeouts, 408, 429, and server errors, a bounded number of times, with exponential backoff and a little random jitter, so that thousands of browsers don’t all retry in the same instant.
Sometimes the connection drops just after storage has already saved the file, so the browser never hears that it worked. Retrying is still safe. The retry writes the same file to the same key, and the server checks what actually landed either way.
A 403 needs a different answer. It can mean the URL expired, the signed headers didn’t match, or a policy refused the request. Retrying it with the same URL will just fail again. Ask the server about the file’s state. If the URL expired, get a fresh one. Otherwise, show the error instead of retrying.
People will want to cancel, too, and to come back to a half-finished batch after closing the tab. On screen, show overall progress and the state of each file. If someone is using a screen reader, though, don’t read every percentage out loud. Hearing “41 percent, 42 percent, 43 percent” for thousands of files is exhausting, so only announce when a file finishes or fails.
Keep that byte-by-byte progress in the browser, too. The database only needs to hear about the moments that matter, like a file being requested, uploaded, or verified.
“Done!”, says the browser
When a file lands, storage answers the browser with a success, and the browser tells your API that the file is done. It’s tempting to take that at face value. But a browser can be wrong, and someone determined can make it lie.
So when a “done” report comes in, the server goes and looks for itself. It makes sure the report comes from the file’s owner, finds the record it saved before signing the URL, and looks at the object in storage, at the key the server chose. Is the file there? Is it the size we expected?
The right size doesn’t prove the bytes arrived intact, though. When that matters, have the browser compute a checksum of the file and send it with the upload. Amazon S3 and Google Cloud Storage, for example, check it on arrival and reject the upload if the bytes don’t match. How that works with signed URLs varies, so check your provider’s docs.
Checking that the file arrived is only half the job. The browser said it was sending a PDF, but that’s just a label it chose. So before anything opens the file for real, check that the bytes look like the type they claim to be, and run whatever scanning your product needs, such as a malware scan.
Some files are actively hostile. A zip bomb, for example, is a tiny archive that expands into an enormous amount of data the moment something unpacks it. Anything that unzips, resizes, or converts uploads needs limits of its own. This kind of inspection can be slow, so it usually runs in the background, on a copy the browser can no longer touch. More on that copy in a moment.
One thing not to lean on here is the ETag that storage hands back, at least not as a checksum. For files uploaded in parts it isn’t the file’s MD5, as AWS’s multipart upload overview (opens in a new tab) explains. It does change whenever the object’s contents change, which turns out to be useful in a moment. It just isn’t a version number.
What if the file changes after the check?
There’s one more gap, and it’s a sneaky one.
A signed URL works until it expires, not until your server has checked the upload. Both R2 and S3 let a URL be used again and again within that window. So someone could upload a harmless file, wait for your server to check it, and then use the same URL to replace it with something else.
The fix is to move the file out of reach. Once the quick checks pass, and the file is there at the size we expected, the server copies it to a place only the server can write to.
There’s still a split second between the check and the copy, and a swap could land right in it. So make the copy conditional on the ETag you checked, and storage will refuse to copy a file that changed in between (R2 lists the conditions it supports (opens in a new tab)). On S3 with versioning turned on, there’s a sturdier option. Record the version ID you checked, which really is a version number, and copy that exact version. If your provider can do neither, run the quick checks again on the copy.
The slower inspection from earlier, the type check and the malware scan, then runs on the copy, and so does everything after it.
Once the copy passes inspection, the server writes it down. The file’s record moves from pending to verified, but only if it was still pending. That way, reporting the same file twice doesn’t verify it twice.
If the file needs processing, there’s one last small trap. Saving “verified” and putting a job on the queue are two separate steps, and the server could crash between them. The usual way around it is to save a note that says “this file needs processing” in the same database transaction as the verified status. A separate worker reads those notes and puts the jobs on the queue, retrying until it works. This pattern is called an outbox.
Two things that bite later
Neither of these changes the design, but both tend to show up the first time you try it for real.
The first is CORS. Your page and your bucket usually live on different domains, so the browser won’t let the upload through unless the bucket says it’s allowed. That permission is a CORS policy on the bucket. Allow your own site, the upload method, and the headers you sign, and nothing more. Keep in mind what CORS does. It tells the browser what it may send and read. It doesn’t decide who’s allowed to upload, so it’s no replacement for any of the checks above. R2 (opens in a new tab), S3 (opens in a new tab), and Cloud Storage (opens in a new tab) each have a page on setting it up.
The second is one enormous file. Everything so far assumes lots of normal-sized files. If a single file is too big for one upload, or needs to survive a dropped connection halfway through, you’d reach for multipart upload instead. The file goes up in numbered parts, and storage stitches them back together at the end. The limits vary by provider. R2, for example, caps a single upload at 5 GiB (opens in a new tab), and allows multipart uploads of just under 5 TiB, in at most 10,000 parts. And on S3, a multipart upload you never finish keeps its parts stored until you complete it or abort it, so it’s worth cleaning those up.
Back to month-end
Let’s go back to that morning, with its 300 people and 600,000 files.
The bytes now travel from each browser straight to object storage, a few files at a time per person. Your API hands out short-lived permissions a page at a time, checks what arrived, and records it. The database takes a write when a file changes state, not for every chunk. And the servers behind the login page are no longer relaying terabytes.
Your API’s share still adds up. 600,000 files still mean 600,000 records, signatures, and checks. If everyone finishes around nine thirty, those “done” reports arrive in a wave. So the server needs a pace of its own, too, checking and processing files as fast as the database can comfortably take them, not as fast as they arrive.
None of this makes the spike go away. Storage still costs money, someone can still try to abuse the upload, and the database still records every file. Keep the pages of URLs small, cap how many uploads each browser runs, expire uploads nobody finished, clean up files nobody claimed, and leave the heavy processing to a queue.
That queue raises one more question. When one customer drops 2,000 files and another drops twelve, whose files get processed first? That’s a problem for another post.