Skip to main content

Linux

Utility for Monitoring Data Transfer

I was performing a website migration, where I had to manually move data from a SQL file on my laptop to a new MySQL database (RDS) (approximately 450MB in size)

I wanted to track the import progress and came to know about the Pipe Viewer (pv) utility.

Pipe viewer, also known as pv, is a terminal-based tool that can be used to monitor the progress of data transfer.

Command Example

pv backup.sql | mysql -h rds.amazonaws.com -u bibinwilson -p my_db

It can be inserted into any normal pipeline between two processes to visually indicate how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

Features
-Visual progress bar
– Estimated time remaining
– Latency

You need to install the utility from the respective package manager.

Tool: Pipe Viewer Utility

How does it track the time?

When data is piped through pv, it reads the input stream and counts the number of bytes transferred. By periodically checking the number of bytes read and the elapsed time, pv can calculate the transfer rate and estimate the time remaining for the transfer to complete.

Here’s a simplified explanation of how pv tracks the progress and estimates the time remaining:

  1. Bytes Transferred: As pv reads data from the input stream, it keeps track of the number of bytes read.
  2. Elapsed Time: pv also keeps track of the elapsed time since it started reading data.
  3. Transfer Rate: By dividing the number of bytes transferred by the elapsed time, pv calculates the transfer rate in bytes per second.
  4. Progress: pv compares the number of bytes transferred with the total size of the input stream (if known) or a specified size (if provided). It calculates the percentage of completion based on the ratio of transferred bytes to the total size.
  5. Estimated Time Remaining: Using the transfer rate and progress, pv estimates the time remaining for the transfer to complete. It calculates the remaining bytes to transfer by subtracting the transferred bytes from the total size. Then, dividing the remaining bytes by the transfer rate provides an estimate of the time remaining.

How accurate is the estimated time remaining?

If the transfer rate fluctuates significantly, the estimate may not be precise.