[HOWTO] Sync 2 Folders

Post tutorials, HOWTO's and other useful resources here.

[HOWTO] Sync 2 Folders

Postby FURRY_NOVA » 21 Sep 2016, 05:01

Intro~
Syncing files has become an important role in todays computing. If it's either keeping 2 devices up-to-date, or backing up and recovering. This small tutorial will teach you some basic methods.

Methods~
There are 4 Linux methods. They are:
  1. rsync Terminal - A command-line built in Porteus Terminal.
    Image
    Here's the command I use for syncing 2 directories. Be aware that the destination will match the source. Even if ya delete and replace one in destination, source will have top priority what the changes will be.
    Code: Select all
    rsync --delete --modify-window=1 --progress --recursive --times --update "/home/guest/Desktop/1/" "/home/guest/Desktop/2/"

    The first directory is the source /home/guest/Desktop/1/.
    The second directory is the destination /home/guest/Desktop/2/.

  2. .desktop Launcher - A method of having a saved double-click button on your desktop to run a stored rsync command.
    Image
    This involves making a .desktop application launcher file. The process is simple. Open a text editor such as Mousepad and copy the following code inside it. Then edit source (A="/home/guest/Desktop/a/") and destination (B="/home/guest/Desktop/b/").When you're happy, save the file as any name you want as long as it ends in ".desktop" without the quotes.
    Code: Select all
    [Desktop Entry]
    Name=Sync
    Exec=/bin/bash -c 'A="/home/guest/Desktop/a/"; B="/home/guest/Desktop/b/"; TXT_GREEN="\033[1;32m"; TXT_RED="\033[1;31m"; TXT_NONE="\033[0m"; echo "Running sync simulation..."; echo -e "${TXT_GREEN}${A}${TXT_NONE} >> ${TXT_RED}${B}${TXT_NONE}"; echo ""; rsync --delete --dry-run --modify-window=1 --progress --recursive --times --update "${A}" "${B}"; echo "Simulation completed."; echo ""; read -rsp "Press any key to perform operation, else close window to abort...\n" -n1 key; echo ""; rsync --delete --modify-window=1 --progress --recursive --times --update "${A}" "${B}"; echo "Operation completed."; echo ""; read -rsp "Press any key to exit...\n" -n1 key;'
    Type=Application
    Icon=emblem-synchronizing
    Comment="Rsync from directory A to B, specified in this launcher."
    Path=
    Terminal=true
    StartupNotify=false

  3. grsync GUI - An application which can be downloaded from USM. It's less that 1mb and requires no dependencies requirement.
    Image

  4. syncthing GUI - Another application. Although I've never used it before. More info can be found here:
    - [HOWTO] Sync 2 Folders: syncthing
    - [HOWTO] Sync 2 Folders
Which is best?~
Personally I used the .desktop Launcher version. Works exactly the same as rsync Terminal which is basically what it is. The grsync is buggy from my experiance and syncthing I hadn't tried.

Help~
You can get a list of all the rsync perimeters usable in the command-line by just entering rsync or man rsync.
Last edited by FURRY_NOVA on 01 Nov 2016, 06:29, edited 2 times in total.
New to Linux but learning. Hoping to help out Porteus community. :)
User avatar
FURRY_NOVA
Black ninja
Black ninja
 
Posts: 81
Joined: 11 Aug 2016, 18:26
Location: United Kingdom
Distribution: Porteus XFCE v3.2 i586

Re: [HOWTO] Sync 2 Folders

Postby Bogomips » 21 Sep 2016, 12:51

Nice presentation. Rsync has ginormous options list, and fact that distilled down to a handful makes it easy to build on..

An enhancement could be running a cron job.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB
Bogomips
Full of knowledge
Full of knowledge
 
Posts: 2239
Joined: 25 Jun 2014, 16:21
Location: London
Distribution: 3.2.2 Cinnamon & KDE5

Re: [HOWTO] Sync 2 Folders

Postby FURRY_NOVA » 29 Sep 2016, 04:10

Update
I've added a Shortcut Method w/ Simulation.

Although I'm sure it should be possible to shorten the code by applying the directories once with variables but my knowledge is lacking in Linux terminal.

Example of my explanation. (Not an actual code example.)
Code: Select all
a="/home/guest/Desktop/1/";
b="/home/guest/Desktop/2/";
/bin/bash -c 'echo "Initializing rsync..."; rsync --delete --modify-window=1 --progress --recursive --times --update a b; echo "Operation completed."; read -rsp $"Press any key to close...\n" -n1 key'
New to Linux but learning. Hoping to help out Porteus community. :)
User avatar
FURRY_NOVA
Black ninja
Black ninja
 
Posts: 81
Joined: 11 Aug 2016, 18:26
Location: United Kingdom
Distribution: Porteus XFCE v3.2 i586

Re: [HOWTO] Sync 2 Folders

Postby mocabilly » 29 Sep 2016, 20:51

How about using syncthing?
Especially when you want to sync between multiple devices/servers/..

Been using it now for a couple of days and I'm just loving it..
Not sure though there is a module for Porteus.. :S
There are only 10 kind of people in this world, those who can read binary and those who can't.
User avatar
mocabilly
White ninja
White ninja
 
Posts: 21
Joined: 12 Jan 2011, 18:53
Location: Belgium
Distribution: Mint-Porteus

Re: [HOWTO] Sync 2 Folders

Postby francois » 30 Sep 2016, 00:03

Un revenant. :shock:
Voltaire: Le mieux est l'ennemi du bien.
User avatar
francois
Contributor
Contributor
 
Posts: 4672
Joined: 28 Dec 2010, 15:25
Location: C'est l'autome, en forêt les arbres d'ici nous en mettent plein la vue: les rouges, jaunes ...
Distribution: kde xfce porteus manjaro kubun

Re: [HOWTO] Sync 2 Folders

Postby FURRY_NOVA » 30 Sep 2016, 00:25

I tried checking it out. It isn't on USM. Also I tried downloading the source but I don't know how to compile it into a xzm module. I tried that src2pkg that comes with the Devel module, even done the "src2pkg --setup" and still get errors trying to "src2pkg /dir_to_source_file". :(
New to Linux but learning. Hoping to help out Porteus community. :)
User avatar
FURRY_NOVA
Black ninja
Black ninja
 
Posts: 81
Joined: 11 Aug 2016, 18:26
Location: United Kingdom
Distribution: Porteus XFCE v3.2 i586

Re: [HOWTO] Sync 2 Folders

Postby brokenman » 01 Oct 2016, 03:46

Mocabilly! Great to see you here!

Gem of an application you posted about. Looks really interesting, kind of how cloud should be.
How do i become super user?
Wear your underpants on the outside and put on a cape.
brokenman
Site Admin
Site Admin
 
Posts: 5343
Joined: 27 Dec 2010, 04:50
Location: Brazil
Distribution: Porteus v3.2rcX all desktops

Re: [HOWTO] Sync 2 Folders: syncthing

Postby Bogomips » 03 Oct 2016, 19:10

SYNCTHING
  • Code: Select all
    USM version:  3.2.0
    DB version:  160320
    Current storage: /home/guest/p10/Por/tmp/usm
    Current config: /etc/usm/usm.conf
    Current platform: Porteus-v3.2rc5
    Current arch: i686
    Current Slackware 14.2
    Current kernel 4.7.2-porteus
    Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
    USM offered to build. No thanks. Via pkgs.org downloaded 14.1 package from slackonly.
  • After making module and activating, checked for outstanding dependencies using modified form of ralcoer's script (Missing dependencies) for just the one bin/lib, ckdeps.sh (non-final version):
    Code: Select all
    #!/bin/bash
    #   03.10.2016 16:04:57
    # Check for outstanding dependencies. Ralcoer's script modified for just one bin/lib.
    # ckdeps.sh   <bin/lib>   [Directory holding bin/lib. Default: executable in /usr/bin]
    ckdeps()
    {
    dir=${2:-/usr/bin};
    if [[ ! -d ${dir} ]]; then
       echo "Parameter is not a directory."
       return 1
    fi
    [[ ! $1 ]] && return 2;
    file=${dir%/}/$1
    if [[ -e  $file ]]; then
        if [ ! -d ${file} ]; then
          elf=`file ${file} | grep ELF`
          if [ "${elf}" != "" ]; then
              missing=`ldd ${file} | grep "not found"`
              if [ "${missing}" != "" ]; then
                echo "-------"
                echo "${file}"
           echo "${missing}"
              fi
          fi
        fi
    else
       echo File $1 not found in $dir; return 1
    fi
    }
    ckdeps $*
    Found no outstanding dependencies for syncthing.
  • Ran in terminal and after what seemed like taking an eternity to make the RSA key on my m/c, Light browser came up with a start page, despite the fact that Palemoon was already up and running.. Checking preferred applications showed Light as browser app, so presumably syncthing had invoked a browser application. Then at a loss how to proceed further.
  • At next effort. when Pomodoro Timer recommended a short break, started up syncthing and gave it some ten minutes to make the RSA key. Upon returning, Light browser was up on start page. This time, having recalled reading in documentation about listening on some local port, inserted a link from terminal output into browser,
    Code: Select all
    guest@porteus:~$ syncthing
    [monitor] 15:01:17 INFO: Starting syncthing
    [start] 15:01:17 INFO: Generating RSA key and certificate for syncthing...
    [2BUIR] 15:01:53 INFO: syncthing v0.11.10 (go1.4.2 linux-386 default) unknown-user@syncthing-builder 2015-06-21 09:45:54 UTC
    [2BUIR] 15:01:53 INFO: My ID: 2BUIRFS-CUGAPJI-W22XN6Q-6G3SDKW-52UWCPK-GE4ANRP-A2F2FOW-XHB5HAX
    [2BUIR] 15:01:53 INFO: No config file; starting with empty defaults
    [2BUIR] 15:01:53 INFO: Edit /home/guest/.config/syncthing/config.xml to taste or use the GUI
    [2BUIR] 15:01:53 INFO: Database block cache capacity 8192 KiB
    [2BUIR] 15:01:53 OK: Ready to synchronize default (read-write)
    [2BUIR] 15:01:53 INFO: Starting web GUI on http://127.0.0.1:8384/
    [2BUIR] 15:01:53 INFO: Loading HTTPS certificate: open /home/guest/.config/syncthing/https-cert.pem: no such file or directory
    [2BUIR] 15:01:53 INFO: Creating new HTTPS certificate
    [2BUIR] 15:01:53 INFO: Generating RSA key and certificate for porteus...
    [2BUIR] 15:01:53 INFO: Completed initial scan (rw) of folder default
    [2BUIR] 15:02:11 INFO: Starting local discovery announcements
    [2BUIR] 15:02:11 INFO: Starting global discovery announcements
    [2BUIR] 15:02:11 INFO: Device 2BUIRFS-CUGAPJI-W22XN6Q-6G3SDKW-52UWCPK-GE4ANRP-A2F2FOW-XHB5HAX is "porteus" at [dynamic]
    [2BUIR] 15:02:11 INFO: API listening on 127.0.0.1:8384
    [2BUIR] 15:02:21 INFO: New UPnP port mapping: external port 31406 to local port 22000.
    and Admin GUI came up:Image
  • Module just 3.2 MB, and syncthing looks like not very demanding on memory, not at least at the start:
    Code: Select all
    guest@porteus:~$ ls -lh syncthing-0.11.10-i486-1_slack.xzm
    -rw-r--r-- 1 root root 3.2M Oct  2 18:25 syncthing-0.11.10-i486-1_slack.xzm

    # Running with LightPalemoon, Nemo, Geany, Terminator:
    guest@porteus:~$ free -m
                  total        used        free      shared  buff/cache   available
    Mem:            879         492          87          80         299         145
    Swap:           644         304         340

    All above+flash plugin-container:
    free -m
                  total        used        free      shared  buff/cache   available
    Mem:            879         496          68          56         314         165
    Swap:           644         396         248
It seems, unlike rsync, a change in one synced folder is reflected in all!
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB
Bogomips
Full of knowledge
Full of knowledge
 
Posts: 2239
Joined: 25 Jun 2014, 16:21
Location: London
Distribution: 3.2.2 Cinnamon & KDE5

Re: [HOWTO] Sync 2 Folders

Postby mocabilly » 12 Oct 2016, 20:56

Memory & processing time will increase with the number of files and sub-directories you sync with Syncthing.
It will take a lot files and subdirectories before it consumes to much of the resources..
There are only 10 kind of people in this world, those who can read binary and those who can't.
User avatar
mocabilly
White ninja
White ninja
 
Posts: 21
Joined: 12 Jan 2011, 18:53
Location: Belgium
Distribution: Mint-Porteus

Re: [HOWTO] Sync 2 Folders: Limiting Case of Two Files

Postby Bogomips » 04 Nov 2016, 02:06

Syncing Two or More Files

Folders could not be synced, as belonging to different architecture. Best could do was to resort to Links or Symlinks. With this approach, one file has to hold the real contents, while all the others are links to it. No danger of accidental deletion, as it will only be a link that is deleted.

Question then arises of where to hold the real contents, best could come up with was to have the file in a directory, difficult to accidentally delete, a root directory.

Creating a Symlink:
Code: Select all
ln  -s  /path/to/file  /path/to/symlink
# Important
ln  -s  <Absolute Path to File>  <Absolute Path to Symlink
Got caught here, as there is no error message output when relative paths given, and in certain cases symlink created, actually works.

Example

  • Salt away Contents
    Code: Select all
    guest@porteus:~$ sudo mkdir /mnt/sda4/safe

    guest@porteus:~$ ls -ld /mnt/sda4/safe/
    drwxr-xr-x 2 root root 48 Nov  3 19:58 /mnt/sda4/safe/
    guest@porteus:~$ sudo chmod a+w /mnt/sda4/safe
    guest@porteus:~$ ls -ld /mnt/sda4/safe/
    drwxrwxrwx 2 root root 48 Nov  3 19:58 /mnt/sda4/safe/

    guest@porteus:~$ touch /mnt/sda4/safe/"The files in this directory are reasonably safe from accidental deletion"

    guest@porteus:~$ ln /mnt/sda4/PaleMoon/pmProfile/contentblock-regex.txt  /mnt/sda4/safe/
    guest@porteus:~$ ls /mnt/sda4/safe/
    The\ files\ in\ this\ directory\ are\ reasonably\ safe\ from\ accidental\ deletion
    contentblock-regex.txt

  • Sync the Files
    Code: Select all
    guest@porteus:~$ mv /mnt/sda1/PaleMoon/pmProfile/contentblock-regex.txt /mnt/sda1/PaleMoon/pmProfile/bak_contentblock-regex.txt
    guest@porteus:~$ ln  -s  /mnt/sda4/safe/contentblock-regex.txt    /mnt/sda1/PaleMoon/pmProfile/

    guest@porteus:~$ ls -l  /mnt/sda1/PaleMoon/pmProfile/contentblock-regex.txt
    lrwxrwxrwx 1 guest users 42 Nov  3 21:26 /mnt/sda1/PaleMoon/pmProfile/contentblock-regex.txt -> /mnt/sda4/safe/contentblock-regex.txt

    guest@porteus:~$ tail -n3  /mnt/sda1/PaleMoon/pmProfile/contentblock-regex.txt
    terraclicks.com
    # 02.11.2016 19:45:30
    liveadexchanger.com
Maybe there's a better way? :unknown:
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB
Bogomips
Full of knowledge
Full of knowledge
 
Posts: 2239
Joined: 25 Jun 2014, 16:21
Location: London
Distribution: 3.2.2 Cinnamon & KDE5


Return to Tutorials



Who is online

Users browsing this forum: No registered users and 2 guests