Uploading files through "shaped" connections without traffic control

Technically shaping is limiting the rate at which packets are sent over a connection. If you want to continue surfing the web or interacting with websites while uploading your files to online storage, you have to find some way to shape your connection to your online storage. Not only will you not be able to interact with websites while uploading files, but your transfer statistics won't be accurate.

There are really only two ways you can shape a connection without traffic control. You can either use a relay that supports delaying packets or find some way to slowdown the rate at which your file is sent.

In any case, the first thing you have to do is determine how much of your bandwidth you want to dedicate to uploading files. Then you have to convert your bandwidth to a unit measurement that is recognizable by the command rsync. When you're ready you can play with the next command.

Here's a one-liner for transferring a file from a remote server via sshd then uploading it to online storage via mount.davfs at directory cloud, and limiting the rate at which the file is sent to 450 Kbps:

rsync --bwlimit=55 --progress $(scp -2 -v root@remote_server:/root/dir/$(n=file_name; echo $n) .; echo $n) cloud/dir/$n

Do you have a suggestion about how to improve this blog? Let's talk about it. Contact me at David.Brenner.Jr@Gmail.com or 720-584-5229.

Comments

Network traffic monitoring in Linux with Python

You can investigate suspicious activity in your network traffic by collecting relevant machine data from your endpoint. You can use the machine data to create your own analysis. Before you start your investigation you will need to determine normal activity on your endpoint. Normal activity is the scope of functionality of the software on your endpoint during periods of low activity and high activity. You will need some kind of software that periodically collects specific machine data from your endpoint like my software developed in Python that's available for free download at https://github.com/davidbrennerjr/server-stats-collector Ingest one or more of the following machine data: Application specific logs from /var/log Raw dumps from sniffing at Layers 2-3 Raw dumps from /proc of kernel data structures Raw dumps of kernel routing tables General system-wide error messages from /var/log/syslog Do you

OpenStack+Ceph as Software-Defined Storage

SDS reduces the costs of the management of growing data stores by decoupling storage management from its hardware to allow for centralized management of cheaper, popular commodity hardware. The example SDS ecosystem uses open source software like OpenStack as a front-end interface on top of Ceph as the resource provider of a RADOS cluster of commodity solid-state drives. OpenStack provides user-friendly wrappers for accessing and modifying underlying Ceph storage. OpenStack comes in the form of distributed microservices with RESTful API's: Block (Cinder), File (Manila), Image (Glance), and Object (Swift). Each microservice can scale-out as a cluster of stand-alone services to accommodate the varying demands of high-growth storage. With OpenStack the underlying Ceph storage can address the block storage needs, file storage needs, image storage needs, and object storage needs of datacenters adopting open source as their new norm in an industry trend for high performace and high a

What are attack vectors?

In the generalized sense an attack vector is a path or means by which a hacker can gain unauthorized access to an endpoint in order to deliver a payload or to facilitate a crime. Attack vectors enable hackers to exploit vulnerabilities in the design of a network through the manipulation of applications and protocols. Attack vectors typically manipulate the software installed in the operating system of an endpoint. Examples of attack vectors are email attachments, pop-up windows, instant messages, service configurations, new software, and firewall modifications. Human ignorance or weaknesses could also be used for engineering attack vectors. For example, users could be fooled into weakening network defenses during times of remote collaboration and file sharing. Anti-virus software and firewalls do provide some defense or block attack vectors to some extent. Some of the mitigation measures used to thwart hackers usage of attack vectors include deep packet inspection, IP source trackers

David Brenner Jr's Random Things

Search This Blog