Skip to main content

SFTP 1.5.1: Reliable Large Transfers

· 7 min read
Tera Insights Team
Tera Insights Team
tiCrypt Team

Sites moving large datasets through SFTP inboxes reported a specific, frustrating pattern: a big transfer would complete, and then the next transfer would fail. Sometimes it took a single multi-gigabyte file, sometimes a directory with thousands of small ones. SFTP 1.5.1 fixes that, and rebuilds the upload pipeline underneath it.

What Was Happening​

The failure had two halves, and they fed each other.

The upload pipeline could crash the service. Files are split into 8 MiB chunks, encrypted, and streamed into the recipient's inbox. The code that assembled those chunks miscounted bytes whenever a client's write request straddled a chunk boundary -- something that happens on every boundary with request sizes that do not divide evenly into 8 MiB. Miscounting led to a chunk being sent before it was complete, and to conditions the service could not recover from. Separately, every chunk allocated a fresh 8 MiB buffer and uploads ran without any limit on how many could be in flight, so a fast client on a good connection could drive memory into the hundreds of megabytes and be terminated by the host.

The service did not always clean up after itself. Transfer state is tracked per session in a local database and cleared when a user's last connection closes. Two connections closing at the same instant -- the normal behavior of graphical SFTP clients, which open several in parallel -- could each conclude the other was still active, and neither would clean up. A restart cleared memory but not the database, and because an inbox reuses the same SFTP username every time, the next transfer collided with the leftovers and was rejected.

Put together: a large transfer could take the service down or leave state behind, and the user's next attempt failed with a permission or write-protection error that had nothing to do with their permissions.

What Changed​

Transfers complete or fail honestly​

Chunk accounting was rewritten. Every byte is placed by its position in the file, chunks are finalized only once their last byte has arrived, and a write that arrives for a region already sent is refused rather than silently dropped.

The more important change is what happens when something does go wrong. Previously an upload failure was logged and forgotten while the client was told the file had been stored. Now a failed chunk fails the file: the SFTP client sees the error on close, the size is not recorded, and the transfer summary counts it as failed. A file that reports success is a file that arrived complete.

Memory is bounded​

Chunk buffers are pooled and reused instead of allocated per chunk, and finished chunks move through a queue of fixed depth. When the queue is full the service stops accepting writes until an upload drains it -- the client slows down instead of the service growing.

Memory per open file is now bounded at roughly 120 MiB regardless of how large the file is or how fast the client sends. This is verified on every build: a test streams a synthetic file of arbitrary size through the real pipeline, checks every uploaded chunk byte-for-byte, and fails if buffer use grows with file size or if any buffer is left outstanding. It has been run at sizes from 64 MiB to 1 GiB with flat memory.

note

Two settings in the [sftp] section of ticrypt-sftp.toml tune this pipeline. The defaults suit most deployments.

upload_queue_depth = 3    # finished chunks that may wait before writes block
upload_workers = 2        # chunks encrypted and uploaded concurrently

Raising them trades memory for throughput on high-latency links to tiCrypt.

Writes no longer wait on the network​

Each write used to block until a full 8 MiB chunk had been encrypted and pushed to tiCrypt, which stalled the client's request pipeline on every chunk boundary. Uploads now run in the background while the client keeps sending. On links where the round trip to tiCrypt is the limiting factor, this is the difference between one chunk per round trip and a continuously filled pipe.

tip

Throughput depends on your network, host, and tiCrypt instance. Measure a representative transfer before and after upgrading rather than assuming a figure.

Sessions clean up exactly once​

The end-of-session cleanup now makes its decision atomically, so parallel connections can no longer both defer to each other. Session state is scoped per transfer rather than per username, and any state left behind by an earlier run is cleared at startup. A transfer that ends badly no longer poisons the next one.

The service also no longer terminates on conditions it previously could not survive, including a failure to accept an incoming connection -- the state that a busy host reaches after heavy transfer activity.

Shutdown waits only when there is something to wait for​

Stopping or restarting the service now checks whether transfers are actually in progress.

At the moment of systemctl stopBehavior
No active SFTP connectionsStops immediately
Transfers in progressWaits up to 30 seconds, exits as soon as they finish
Still running at 30 secondsCloses remaining connections, flushing buffered data and writing transfer summaries

Both the command line and the log report which path was taken, with progress while waiting:

Received terminated: graceful shutdown started, 2 SFTP connection(s) in 1 transfer session(s)
still active. Shutting down in 30s, or sooner if they finish
Graceful shutdown: still waiting for 2 SFTP connection(s) in 1 session(s); 25s remaining
Graceful shutdown: all transfers finished after 12s. Shutdown complete

A restart during an upload no longer abandons transfer state, which was itself a source of the failure pattern above.

Web session lifetimes behave as documented​

The extend_by and max_lifespan settings that govern tiCrypt user sessions on the inbox management API were parsed but never applied; sessions expired a fixed 30 minutes after sign-in regardless of activity. Sessions now extend while in use and stop at max_lifespan, and contradictory lifetime settings are rejected at startup instead of being silently ignored.

Supporting changes​

  • Database. The local metadata store uses write-ahead logging, which removes most of the disk synchronization cost per file. Directory-heavy transfers with thousands of small files benefit most.
  • Diagnostics. An optional pprof_listen address exposes Go profiling endpoints for support investigations. It is disabled by default and should stay bound to loopback when enabled.
  • Service unit. The systemd unit now allows time for a graceful stop and raises the file descriptor limit for deployments with many parallel connections.

Upgrading​

  1. Install the 1.5.1 package from your configured tiCrypt repository.
  2. Review ticrypt-sftp.toml. No changes are required; the new [sftp] upload settings and pprof_listen are optional and default to previous behavior.
  3. Restart the service. If a transfer is in progress it will be given the grace period described above.
caution

If a previous version left transfer state behind, users may still have been seeing failures on repeat uploads to the same inbox. Version 1.5.1 clears that state at startup, so the first restart after upgrading resolves it. No manual database cleanup is needed.

note

Inbox behavior, credentials, and the sender-facing workflow are unchanged. External collaborators use the same SFTP clients and the same credentials page as before.