ICO wiki - User contributions [en]

User:Eocakovs

2017-05-26T04:56:53Z

Eocakovs: /* Prototype */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

==Value Chain Analysis==

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

==IT SWOT==

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no direct competitors
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organised and motivated
| Small company, not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer skepticism - might not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

==Requirements==
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

==User Stories==
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

==Data Flow Diagrams==

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]

==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE1.png |1240px|]]

==Architectural Model==
[[File:Architectural_model_WPE.png |800px|]]

==UI - User Experience==
The main goal is to create an application that would be easily operated by the end user.
Since the application is mainly meant for the average home user, it cannot require any specific knowledge about computers, or computer related terminology.
All aformentioned points (additionally see non-functional requirements) need to be kept in mind when designing the UI.

'''Measurable Key Performance Indicators'''

* Success rate (how many of the user requests come back with a positive responce. i.e. - if a user requests for additional app info, can we provide that information, or will the user get back nothing).
* Time on task (e.g. how long does it take for the user to upgrade from standard user to premium user. Does it take too much time, and is the process too complicated).
* Amount of installations.
* Time user spends in app (active time).

'''User feedback'''

* In-app feedback tab (option to select how satisfied the user is with the program in a 1-10 point scale, and a comment box)
* Post service survey. For instance after a user upgrades from standard user to premium user, an email will be sent out with questions about the process and an option to leave feedback and suggestions.
* Email survey. An email will be sent out with a user satisfaction survey, asking users to rate different parts of the process.
* In-app survey before uninstalling/downgrading the WPE (With in-app monetary incentive).

==Constraints==
* Competitors (like 'Revo uninstaller' / Windows native Store)
* Investors / Venture Capital
* Performance (DB / Network / Program Design)
* Balance of team (developers vs testers)
* Time
* Unforseeable constraints we have to deal with using Agile Approach

==Risks and Countermeasures==
{| class="wikitable"
|-
! Hackers and data Breaches
! Hardware failure
! Power outages
! Force majeures
! Software Issues
! User errors
|-
|
* Secure software practices
* Secure network architecture
* Inhouse team training
|
* Backup hardware
* Cloud / Off-site duplication
|
* UPS
* Power generators
* Backup servers
|
* Correct contracts with customer
|
* Increase hiring requirements
* Hire Testers / Debuggers
* Educate current developers
|
* Create intuitive user experience
|}
==Prototype==
[[File:Prototype_wpe.JPG]]

==Recommendations and Conclusion==
Our group identified a need for an information system that would explain to users applications that either are or could be installed. However after careful analysis we have to come to the conclusion that the application will become obsolete due to an increasing push by Microsoft for Windows 10+ and their native Store. For this application to have a future it should integrate with the native Windows store. Therefore we do not proceed to full development and await further action by Microsoft until project becomes viable.

It is suitable for agile since we can identify Minimum Viable Product requirements
but due to constantly evolving application market and Windows platform,
we cannot create a full analysis which lists all requirements from the beginning.

File:Prototype wpe.JPG

2017-05-26T04:56:35Z

Eocakovs:

User:Eocakovs

2017-05-26T04:50:57Z

Eocakovs: /* IT SWOT */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

==Value Chain Analysis==

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

==IT SWOT==

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no direct competitors
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organised and motivated
| Small company, not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer skepticism - might not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

==Requirements==
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

==User Stories==
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

==Data Flow Diagrams==

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]

==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE1.png |1240px|]]

==Architectural Model==
[[File:Architectural_model_WPE.png |800px|]]

==UI - User Experience==
The main goal is to create an application that would be easily operated by the end user.
Since the application is mainly meant for the average home user, it cannot require any specific knowledge about computers, or computer related terminology.
All aformentioned points (additionally see non-functional requirements) need to be kept in mind when designing the UI.

'''Measurable Key Performance Indicators'''

* Success rate (how many of the user requests come back with a positive responce. i.e. - if a user requests for additional app info, can we provide that information, or will the user get back nothing).
* Time on task (e.g. how long does it take for the user to upgrade from standard user to premium user. Does it take too much time, and is the process too complicated).
* Amount of installations.
* Time user spends in app (active time).

'''User feedback'''

* In-app feedback tab (option to select how satisfied the user is with the program in a 1-10 point scale, and a comment box)
* Post service survey. For instance after a user upgrades from standard user to premium user, an email will be sent out with questions about the process and an option to leave feedback and suggestions.
* Email survey. An email will be sent out with a user satisfaction survey, asking users to rate different parts of the process.
* In-app survey before uninstalling/downgrading the WPE (With in-app monetary incentive).

==Constraints==
* Competitors (like 'Revo uninstaller' / Windows native Store)
* Investors / Venture Capital
* Performance (DB / Network / Program Design)
* Balance of team (developers vs testers)
* Time
* Unforseeable constraints we have to deal with using Agile Approach

==Risks and Countermeasures==
{| class="wikitable"
|-
! Hackers and data Breaches
! Hardware failure
! Power outages
! Force majeures
! Software Issues
! User errors
|-
|
* Secure software practices
* Secure network architecture
* Inhouse team training
|
* Backup hardware
* Cloud / Off-site duplication
|
* UPS
* Power generators
* Backup servers
|
* Correct contracts with customer
|
* Increase hiring requirements
* Hire Testers / Debuggers
* Educate current developers
|
* Create intuitive user experience
|}
==Prototype==

==Recommendations and Conclusion==
Our group identified a need for an information system that would explain to users applications that either are or could be installed. However after careful analysis we have to come to the conclusion that the application will become obsolete due to an increasing push by Microsoft for Windows 10+ and their native Store. For this application to have a future it should integrate with the native Windows store. Therefore we do not proceed to full development and await further action by Microsoft until project becomes viable.

It is suitable for agile since we can identify Minimum Viable Product requirements
but due to constantly evolving application market and Windows platform,
we cannot create a full analysis which lists all requirements from the beginning.

User:Eocakovs

2017-05-26T04:49:30Z

Eocakovs: /* IT SWOT */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

==Value Chain Analysis==

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

==IT SWOT==

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no direct competitors
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organized and motivated
| Small company not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer scepticisim - do not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

==Requirements==
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

==User Stories==
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

==Data Flow Diagrams==

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]

==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE1.png |1240px|]]

==Architectural Model==
[[File:Architectural_model_WPE.png |800px|]]

==UI - User Experience==
The main goal is to create an application that would be easily operated by the end user.
Since the application is mainly meant for the average home user, it cannot require any specific knowledge about computers, or computer related terminology.
All aformentioned points (additionally see non-functional requirements) need to be kept in mind when designing the UI.

'''Measurable Key Performance Indicators'''

* Success rate (how many of the user requests come back with a positive responce. i.e. - if a user requests for additional app info, can we provide that information, or will the user get back nothing).
* Time on task (e.g. how long does it take for the user to upgrade from standard user to premium user. Does it take too much time, and is the process too complicated).
* Amount of installations.
* Time user spends in app (active time).

'''User feedback'''

* In-app feedback tab (option to select how satisfied the user is with the program in a 1-10 point scale, and a comment box)
* Post service survey. For instance after a user upgrades from standard user to premium user, an email will be sent out with questions about the process and an option to leave feedback and suggestions.
* Email survey. An email will be sent out with a user satisfaction survey, asking users to rate different parts of the process.
* In-app survey before uninstalling/downgrading the WPE (With in-app monetary incentive).

==Constraints==
* Competitors (like 'Revo uninstaller' / Windows native Store)
* Investors / Venture Capital
* Performance (DB / Network / Program Design)
* Balance of team (developers vs testers)
* Time
* Unforseeable constraints we have to deal with using Agile Approach

==Risks and Countermeasures==
{| class="wikitable"
|-
! Hackers and data Breaches
! Hardware failure
! Power outages
! Force majeures
! Software Issues
! User errors
|-
|
* Secure software practices
* Secure network architecture
* Inhouse team training
|
* Backup hardware
* Cloud / Off-site duplication
|
* UPS
* Power generators
* Backup servers
|
* Correct contracts with customer
|
* Increase hiring requirements
* Hire Testers / Debuggers
* Educate current developers
|
* Create intuitive user experience
|}
==Prototype==

==Recommendations and Conclusion==
Our group identified a need for an information system that would explain to users applications that either are or could be installed. However after careful analysis we have to come to the conclusion that the application will become obsolete due to an increasing push by Microsoft for Windows 10+ and their native Store. For this application to have a future it should integrate with the native Windows store. Therefore we do not proceed to full development and await further action by Microsoft until project becomes viable.

It is suitable for agile since we can identify Minimum Viable Product requirements
but due to constantly evolving application market and Windows platform,
we cannot create a full analysis which lists all requirements from the beginning.

User:Eocakovs

2017-05-25T13:20:19Z

Eocakovs: /* Architectural Model */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

==Value Chain Analysis==

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

==IT SWOT==

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no other popular program like this
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organized, good
| Small company not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer scepticisim - do not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

==Requirements==
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

==User Stories==
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

==Data Flow Diagrams==

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]

==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE.png |1240px|]]

==Architectural Model==
[[File:Architectural_model_WPE.png |800px|]]

==UI - User Experience==
The main goal is to create an application that would be easily operated by the end user.
Since the application is mainly meant for the average home user, it cannot require any specific knowledge about computers, or computer related terminology.
All aformentioned points (additionally see non-functional requirements) need to be kept in mind when designing the UI.

'''Measurable Key Performance Indicators'''

* Success rate (how many of the user requests come back with a positive responce. i.e. - if a user requests for additional app info, can we provide that information, or will the user get back nothing).
* Time on task (e.g. how long does it take for the user to upgrade from standard user to premium user. Does it take too much time, and is the process too complicated).
* Amount of installations.
* Time user spends in app (active time).

'''User feedback'''

* In-app feedback tab (option to select how satisfied the user is with the program in a 1-10 point scale, and a comment box)
* Post service survey. For instance after a user upgrades from standard user to premium user, an email will be sent out with questions about the process and an option to leave feedback and suggestions.
* Email survey. An email will be sent out with a user satisfaction survey, asking users to rate different parts of the process.
* In-app survey before uninstalling/downgrading the WPE (With in-app monetary incentive).

==Constraints==
* Competitors (like 'Revo uninstaller' / Windows native Store)
* Investors / Venture Capital
* Performance (DB / Network / Program Design)
* Balance of team (developers vs testers)
* Time
* Unforseeable constraints we have to deal with using Agile Approach

==Risks and Countermeasures==
{| class="wikitable"
|-
! Hackers and data Breaches
! Hardware failure
! Power outages
! Force majeures
! Software Issues
! User errors
|-
|
* Secure software practices
* Secure network architecture
* Inhouse team training
|
* Backup hardware
* Cloud / Off-site duplication
|
* UPS
* Power generators
* Backup servers
|
* Correct contracts with customer
|
* Increase hiring requirements
* Hire Testers / Debuggers
* Educate current developers
|
* Create intuitive user experience
|}
==Prototype==

==Recommendations and Conclusion==
Our group identified a need for an information system that would explain to users applications that either are or could be installed. However after careful analysis we have to come to the conclusion that the application will become obsolete due to an increasing push by Microsoft for Windows 10+ and their native Store. For this application to have a future it should integrate with the native Windows store. Therefore we do not proceed to full development and await further action by Microsoft until project becomes viable.

It is suitable for agile since we can identify Minimum Viable Product requirements
but due to constantly evolving application market and Windows platform,
we cannot create a full analysis which lists all requirements from the beginning.

User:Eocakovs

2017-05-25T12:45:19Z

Eocakovs: /* Architectural Model */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

==Value Chain Analysis==

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

==IT SWOT==

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no other popular program like this
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organized, good
| Small company not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer scepticisim - do not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

==Requirements==
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

==User Stories==
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

==Data Flow Diagrams==

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]

==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE.png |1240px|]]

==Architectural Model==
[[File:Architectural_model_WPE.png]]

==UI - User Experience==
The main goal is to create an application that would be easily operated by the end user.
Since the application is mainly meant for the average home user, it cannot require any specific knowledge about computers, or computer related terminology.
All aformentioned points (additionally see non-functional requirements) need to be kept in mind when designing the UI.

'''Measurable Key Performance Indicators'''

* Success rate (how many of the user requests come back with a positive responce. i.e. - if a user requests for additional app info, can we provide that information, or will the user get back nothing).
* Time on task (e.g. how long does it take for the user to upgrade from standard user to premium user. Does it take too much time, and is the process too complicated).
* Amount of installations.
* Time user spends in app (active time).

'''User feedback'''

* In-app feedback tab (option to select how satisfied the user is with the program in a 1-10 point scale, and a comment box)
* Post service survey. For instance after a user upgrades from standard user to premium user, an email will be sent out with questions about the process and an option to leave feedback and suggestions.
* Email survey. An email will be sent out with a user satisfaction survey, asking users to rate different parts of the process.
* In-app survey before uninstalling/downgrading the WPE (With in-app monetary incentive).

==Constraints==
* Competitors (like 'Revo uninstaller' / Windows native Store)
* Investors / Venture Capital
* Performance (DB / Network / Program Design)
* Balance of team (developers vs testers)
* Time
* Unforseeable constraints we have to deal with using Agile Approach

==Risks and Countermeasures==
{| class="wikitable"
|-
! Hackers and data Breaches
! Hardware failure
! Power outages
! Force majeures
! Software Issues
! User errors
|-
|
* Secure software practices
* Secure network architecture
* Inhouse team training
|
* Backup hardware
* Cloud / Off-site duplication
|
* UPS
* Power generators
* Backup servers
|
* Correct contracts with customer
|
* Increase hiring requirements
* Hire Testers / Debuggers
* Educate current developers
|
* Create intuitive user experience
|}
==Prototype==

==Recommendations and Conclusion==
Our group identified a need for an information system that would explain to users applications that either are or could be installed. However after careful analysis we have to come to the conclusion that the application will become obsolete due to an increasing push by Microsoft for Windows 10+ and their native Store. For this application to have a future it should integrate with the native Windows store. Therefore we do not proceed to full development and await further action by Microsoft until project becomes viable.

It is suitable for agile since we can identify Minimum Viable Product requirements
but due to constantly evolving application market and Windows platform,
we cannot create a full analysis which lists all requirements from the beginning.

File:Architectural model WPE.png

2017-05-25T12:44:32Z

Eocakovs:

User:Eocakovs

2017-05-25T07:41:59Z

Eocakovs: /* ERD */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

====Value Chain Analysis====

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

====IT SWOT====

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no other popular program like this
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organized, good
| Small company not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer scepticisim - do not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

===Requirements===
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

===User Stories===
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

====Data Flow Diagrams====

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]
==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE.png |1240px|]]

==Architectural Model==

==UI - User Experience==

==Constraints==

==Risks and Countermeasures==

==Recommendations and Conclusion==

User:Eocakovs

2017-05-25T07:39:40Z

Eocakovs: /* ERD */

= I701 - Information System Analysis =

== Group Project ==

====Members====
* Ardi Vaba
* Pascal Tietjen
* Peep Kuulme
* Nika Ptskialadze
* Erik Ocakovskih
* Frank Korving

====Organization====
WPE (Windows Programs Explained)

'''Daily Operations:''' 
We are a software development company. Our current major project consists of providing and supporting an application with simple interface to analyze currently installed programs on the system. 

'''Example Information Systems:''' 
Accounting Information System 
Customer Support Information System 
Sales Information System 
Developer Daily-Work Information System 
Transaction Processing Information System 

====Value Chain Analysis====

'''Primary Activities:''' 
''Direct''
* Software Development [Getting the software to working state]
* Software Delivery [Getting the product to the market]
* Sales [Selling a product to current customers and new customer aquisition]
* Marketing [Providing product information to target group]
''Indirect''
* Scheduling [Optimizing work and workload assignments]
* Support / Service [Maintenance of existing software and customer assistance]
* Marketing [Customer Management]
''Quality Assurance''
* Testing [Functionality verification of the developed working software]

 
'''Support Activities:''' 
''Direct''
* Infrastructure [Providing Legal, Administrative and Accounting support]
* Procurement [Acquisition of external services]
''Indirect''
* HR Management [Human Resource Management]
 

====IT SWOT====

{| class="wikitable"
|-
! Strength
! Weakness
! Opportunities
! Threats
|-
| Innovative - no other popular program like this
| No brand recognition
| Globally extending
| Larger company might sprint past with a similar program
|-
| Strong team - well organized, good
| Small company not a lot of financial backup
| Expand the team for faster development
| Sustainable financial backing
|-
| Competitively priced
| Development slow due to the size of the company
| Targeting industries
| Customer scepticisim - do not want to give access to scan their computer
|-
| User-friendly interface
| Organic growth - no outside funding
|
|
|}

====Improvements====

'''Problems:''' 
* Application Information
** Due to lack of program information / Methods of updating
** Trustworthiness / Unknown Sources
** Customer Perception of the Products [Solution should provide: Transparency for customers / Awareness of Application]
* Market access / interactions

====Short Information System Description====

Information System will provide Windows OS users with detailed information on their installed packages and applications. Provided information will include trustworthiness / rating / availability / checksum / developer details of the application. System will incentivize users and developers to submit descriptions of applications, which would go through a review process before publication. Companies can become verified by us through a paid application verification process thereby granting "trust-ability" to these applications and their developers. Users can subscribe to a live-scanning service which will notify them when any installed packages are compromised or need to be updated.

===Requirements===
====Functional====
''''Users''''
* Users should be able to subscribe to paid services
* Users should be able to scan their computer and check for installed programs
* Users should be able to gain information on these installed programs through us (WPE)
* Users should be able to delete the application
* Users should be able to request suggestions on alternative
* Users should be able to search for similar applications
* Users should be able to search for any application
* Users should be able to submit their rating/opinion to be reviewed by us

''''Companies''''
* Companies should be able to apply for verification
* Companies should be able to submit their own description after verification

====Non-Functional====
* Application shall be operable in offline environment
* All the application details shall be as transparent to user as possible
* Every aspect of the software shall be testable
* User experience of the software shall be as responsive as possible
* User privacy shall be completely protected
* Documentation of software shall be easily understood by new developers
* Backend infrastructure should be scalable
* Software shall be safe to operate by non technical people
* Application shall be intuitively operable by 70 year old with minimal technical knowledge

===User Stories===
* User needs to find specific information on an installed application, enters name into search bar and presses search-button.
* User finds application, selects it and presses view detailed information button.
* User sees no specific details about application and presses "request additional information"-button.
* User looks through provided information, wants to remove the application and presses the "uninstall"-button, confirmation dialogue appears and user confirms his decision.
* User wants to find possible similar applications, presses "similar apps"-button.
* Premium user sees notification presses it and opens up a tab with an active analysis of host system.
* Non-premium user wants to get premium-features, switches to premium-features section which displays features and payment information.
* User selects preferred amount and payment method from premium-features section which opens up an external browser with preferred payment method's website.
* User wants to change how application updates, opens the update options from the settings menu and selects the "ask-for-confirmation-before-update" method.
* User wants to install a program and check whether the developer is verified or not so searches for the verification signature in the application details.
* User wants to leave feedback on specific application, clicks "leave feedback" button in the application information details tab that opens up a text-box, the user enters feedback and presses submit button.

====Use-Case High Level Overview====
[[File:High-lvl-overview-3.png]]

====Data Flow Diagrams====

=====Context diagram=====
[[File:Context diagram.png]]

=====Lvl0-DFD=====
[[File:Lvl0-DFD.png]]

=====Lvl1-DFD=====
[[File:DFD_lvl1_process1-v6.png]]
==Object Oriented Approach==
===Objects===
{| class="wikitable"
|-
! Object
! Class
! Identifier
! Actions
! States
|-
| Bob
| User
| E-mail
| Submit requests / Access Premium Features / Inspect Installed Applications / Give Feedback / Submit Descriptions on Applications
| Verified / Unverified / Registered / Unregistered / Premium / Standard
|-
| Notepad
| Application
| Name + Version
| Text Editing
| Installed / Uninstalled / Running
|-
| Peep
| Development Team
| Company ID
| Verify Application Descriptions and Issue updates
| Working / On Holiday
|-
| Microsoft
| Application Owners
| Company Name
| Submit detailed on owned Application
| Verified/Unverified/Bankrupt
|-
| Request detail on Notepad
| Request
| Unique Request ID
| Query internal App database / Query Windows / Contact server
| Pending / Idle / Denied / Accepted / Finalized
|-
| Report on Notepad
| Report
| Unique Report ID
| Return application detailed information
| Delivery Origin/Successful/Failed
|}

===Actors===
* Customer -> Obtain detailed app information / Pay subscription / (Un-)Install application / Find applications
* Developer -> Develop features / write code / Analyze / Design
* Application Owner -> Get verified / Provide information on Application(s) / Gain users

===Scenario===
1. As a customer I want to order premium features.

Standard User wants to buy premium features, clicks on the premium feature section, enters required client details in fields, clicks on make payment button, redirected to payment site where payment finalization is taken care of, verification email sent, order details saved in database.

2. As a Premium User I want to see alternative applications

Premium User wants to see alternative applications, selects the relevant application, clicks on the search for alternatives button, gets a list of alternatives application.

===ERD===
[[File:ERD_WPE.png]]

==Architectural Model==

==UI - User Experience==

==Constraints==

==Risks and Countermeasures==

==Recommendations and Conclusion==

File:ERD WPE.png

2017-05-25T07:39:08Z

Eocakovs:

File:ERD.png

2017-05-25T07:37:52Z

Eocakovs: Eocakovs uploaded a new version of File:ERD.png

File:DFD lvl1 process1-v6.png

2017-05-19T11:22:09Z

Eocakovs:

User:Eocakovs

2017-05-19T11:21:19Z

Eocakovs: /* Lvl1-DFD */

User:Eocakovs

2017-05-19T11:20:22Z

Eocakovs: /* Lvl1-DFD */

File:DFD lvl1 process1-v5.png

2017-05-19T11:19:33Z

Eocakovs: Eocakovs uploaded a new version of File:DFD lvl1 process1-v5.png

File:DFD lvl1 process1-v5.png

2017-05-19T11:19:03Z

Eocakovs: Eocakovs uploaded a new version of File:DFD lvl1 process1-v5.png

File:DFD lvl1 process1-v5.png

2017-05-19T11:18:36Z

Eocakovs: Eocakovs uploaded a new version of File:DFD lvl1 process1-v5.png

User:Eocakovs

2017-05-19T11:17:31Z

Eocakovs: /* Lvl1-DFD */

File:DFD lvl1 process1-v5.png

2017-05-19T11:17:01Z

Eocakovs: Eocakovs uploaded a new version of File:DFD lvl1 process1-v5.png

User:Eocakovs

2017-05-18T19:37:23Z

Eocakovs:

File:DFD lvl1 process1-v5.png

2017-05-18T19:36:07Z

Eocakovs:

Operating systems

2017-05-08T09:41:19Z

Eocakovs: /* Students share = */

=Lecturer=

[https://wiki.itcollege.ee/index.php/User:Edmund#in_English Edmund Laugasson]

=Intro=
All subject related infotmation will be put up on Wiki page, due to the possibility to have access to the materials even after the subject has concluded. Materials, such as tests, lectures and links to additional materials, will remain available throughout the subject teaching period.

=Aim of this course=

The aim of this course is to introduce the basics of operating systems and IT system life cycle from the viewpoint of the IT system administrator of operating systems. This subject provides hands-on skills needed to complete other field specific subjects in the curriculum.

Lectures give a theoretical background and the labs give hands-on skills on the same topic using Ubuntu Linux Server.

'''This subject is oriented on hands-on practical assignments to compliment the theoretical side of the subject.'''

Learning outcome 1:

A student who has completed the subject is able to perform the most common administrative tasks (user management, software management, disk usage, process management) in at least one of the most popular operating system on a server.

Learning outcome 2:

A student who has completed the subject understands and is able to explain orally the basic concepts of operating systems and its security aspects.

Learning outcome 3:

The student is able to document an operating system's service from an IT systems administrator's viewpoint.

=Portfolio=
Please fulfill [https://goo.gl/forms/BSEkc2NAoFagQ16R2 the following form].

=Deadlines for assignments=

'''03.04.2017''' - Submission of wiki article's topic

'''02.05.2017''' - lab works defended

'''08.05.2017 23:59''' - Submission of wiki article

'''09.05.2017''' - Pre practical test for students, who have done all of their labs

'''16.05.2017''' - Last option to defend lab work

'''23.05.2017''' - Practical test

All dates are inclusive.

=Instant messaging=
Public chat for any subject related questions can be asked at the channel ''#eitc-osadmin:matrix.org'' and client software: [https://riot.im Riot.im].

[https://riot.im/app Launch Riot here in browser].

=Course materials=

* [http://enos.itcollege.ee/~edmund/osadmin/eng/ Course materials can be found here.]
* [https://echo360.e-ope.ee/ess/portal/section/4ef8a128-857f-4d6c-a532-733f394f70dc recordings are available here]

==Links==
* [https://www.codecademy.com/courses/learn-the-command-line learn the command line]
* [http://upload.itcollege.ee/edmund/ some VMs, ISOs]
* [http://www.tecmint.com/10-useful-free-linux-ebooks-for-newbies-and-administrators/ 10 Free Linux Administration e-books]
* [https://www.youtube.com/playlist?list=PLmbPuZ0NsyGS8ef6zaHd2qYylzsHxL63x Introduction to operating systems (videos)]
* [https://en.wikipedia.org/wiki/Operating_system Operating systems (wikipedia article)]
* [http://enos.itcollege.ee/~edmund/osadmin/materials/ additional OSadmin materials]
* [http://enos.itcollege.ee/~edmund/materials/ additional materials]
* [http://debian-handbook.info/browse/stable/ Debian Administrator's Handbook]
* [https://help.ubuntu.com/lts/serverguide/ Ubuntu Server Guide]
* [https://linuxjourney.com/ Linux Journey]
*'''[http://enos.itcollege.ee/~edmund/materials/links.html Linux and other free software related links]'''
* [http://enos.itcollege.ee/~edmund/materials/news.html Linux news, magazines]

==Finding help==
* https://help.ubuntu.com/ - official documentation
* https://ubuntu-manual.org/

Also some searches:

* https://www.startpage.com/do/search?q=ubuntu+tutorial
* https://www.startpage.com/do/search?q=ubuntu+howto
* https://www.startpage.com/do/search?q=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Also I encourage to use YouTube which is full of videotutorials.

* https://www.youtube.com/results?search_query=ubuntu+tutorial
* https://www.youtube.com/results?search_query=ubuntu+howto
* https://www.youtube.com/results?search_query=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Certainly all the information might not be trustful but you can test at first in virtual machine if you have doubts. Also ask from instant messenger ([[#Instant messaging|also current course has one]]), IRC-channels. If you search https://www.startpage.com/do/search?q=ubuntu+irc+channel then you will find them. You will also find IRC client howtos.

Usually if to search "problem name or error message + operating system name, version" (without quotation marks) then you will find results (e.g.: ''bluetooth ubuntu 16.04''). Usually one of the findings will solve a problem or at least give hints what to do and possible workarounds.

Also using different search engines might give different results. In addition of [https://www.google.com/ Google], you can try also [https://www.startpage.com/ Startpage], [https://duckduckgo.com/ DuckDuckGo] etc - https://en.wikipedia.org/wiki/List_of_search_engines

=Lab works=
Here are some introductory lab tasks available. Labs for assessment are coming into I-Tee virtual laboratory system.

Please check also [https://enos.itcollege.ee/~edmund/osadmin/eng/labs/ log about lab works].

==Lab 0==

Installing Ubuntu Server LTS

'''Introduction to Unix command line''' (cd, ls, cat, full path, relative path etc)

==Lab 1==

'''Managing users''' (adduser, addgroup, passwd, /etc/passwd, /etc/shadow)

1) Create a user sysadmin

2) Add a new group devops and add a the user sysadmin to a previously created group.

3) Divert the user sysadmin's password hash via cowsay to a file called hash.txt.

4) Lock the user sysadmin and be ready to show me the indication of the user being locked.

5) Change the user's current home directory into /home/unknown so that the files will also be moved to the new location.

'''Managing files''' (mkdir, cp, mv, rm, touch, nano, less, chmod, chown, rwx, 644 etc)

1) Create a folder march in root user directory and for every march day a subfolder with a name day1, day2, day3 … day31. (Example: /root/march/day1 or /root/march/day2 etc)

2) Modify the march folder owner so that it will be student and the new group audio.

3) Modify the march folder's and its subfolders so that the user can do anything, group can do ls in the folder and cd into it and others can't do anything with it.

4) Create a hard link called network to a file /etc/network/interfaces

5) Copy /var/log directory into march folder so that the timestamp and user info will be preserved.

'''Processes and environment variables''' (kill, using directing input/output/error: |, <, >, >>; env, PATH, HOME etc)

1) Divert the list with the student user's groups via cowsay into a fail studgroup.txt.

2) Create a environment variable called MYHOME that has the value of the system's HOME environment variable. (Hint: you have tu use variable symbol here!)

3) Send 2 htop's to the background and be ready to present how you send a kill signal to the first htop by job number and term signal to the second htop by a process number.

4) Create an alias called bye that logs you out of the terminal. Make this alias permanent.

5) Execute a programm called espdiff and diver the standardoutput to a file called okay.txt and the standard error to a file called notokay.txt.

'''Managing software''' (installation, updating, deleting, apt and dpkg utils)

==Lab 2==

'''Managing disks by creating partitions''' (fdisk, mkfs, blkid, mount, umount)

'''Managing swap''' (mkswap, swapon, swapoff)

=Practical test=

==2016==

[https://docs.google.com/document/d/1FGZcqmQQDF1l32uPUJ6n8x2Tc4gK8nuxS-C9esgRqaQ/edit?usp=sharing First practical test 10th of May 2016]

[https://docs.google.com/document/d/1ZCqOOMkx0dwP0QXLIK_yk_08a8whfJmQbYR1mAoSh7M/edit Second pracical test 24th of May 2016]

=Exam=

==2016==

[https://docs.google.com/document/d/1ofiylCw9YAS8_S9YHEc8cZOvEfCfMfs2wDoc44eDyCU/edit?usp=sharing Practical exam]

[https://docs.google.com/document/d/1gkEDb1g1em9UGhj9n_LIwnhp17gY85U9aPtMfGk56_8/edit# Topics] of the oral exam in Spring 2016

=Wiki article=

'''[[OSadmin_wiki_article|for further information please look here]]'''

=Former materials=
* [[Operating systems 2016]]

= Students share =
[https://kobra.io/#/e/-KjbPHJ0sBTp6Q3QLoIx] Bad documentation

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Operating systems

2017-05-08T09:38:27Z

Eocakovs: /* Students share = */

=Lecturer=

[https://wiki.itcollege.ee/index.php/User:Edmund#in_English Edmund Laugasson]

=Intro=
All subject related infotmation will be put up on Wiki page, due to the possibility to have access to the materials even after the subject has concluded. Materials, such as tests, lectures and links to additional materials, will remain available throughout the subject teaching period.

=Aim of this course=

The aim of this course is to introduce the basics of operating systems and IT system life cycle from the viewpoint of the IT system administrator of operating systems. This subject provides hands-on skills needed to complete other field specific subjects in the curriculum.

Lectures give a theoretical background and the labs give hands-on skills on the same topic using Ubuntu Linux Server.

'''This subject is oriented on hands-on practical assignments to compliment the theoretical side of the subject.'''

Learning outcome 1:

A student who has completed the subject is able to perform the most common administrative tasks (user management, software management, disk usage, process management) in at least one of the most popular operating system on a server.

Learning outcome 2:

A student who has completed the subject understands and is able to explain orally the basic concepts of operating systems and its security aspects.

Learning outcome 3:

The student is able to document an operating system's service from an IT systems administrator's viewpoint.

=Portfolio=
Please fulfill [https://goo.gl/forms/BSEkc2NAoFagQ16R2 the following form].

=Deadlines for assignments=

'''03.04.2017''' - Submission of wiki article's topic

'''02.05.2017''' - lab works defended

'''08.05.2017 23:59''' - Submission of wiki article

'''09.05.2017''' - Pre practical test for students, who have done all of their labs

'''16.05.2017''' - Last option to defend lab work

'''23.05.2017''' - Practical test

All dates are inclusive.

=Instant messaging=
Public chat for any subject related questions can be asked at the channel ''#eitc-osadmin:matrix.org'' and client software: [https://riot.im Riot.im].

[https://riot.im/app Launch Riot here in browser].

=Course materials=

* [http://enos.itcollege.ee/~edmund/osadmin/eng/ Course materials can be found here.]
* [https://echo360.e-ope.ee/ess/portal/section/4ef8a128-857f-4d6c-a532-733f394f70dc recordings are available here]

==Links==
* [https://www.codecademy.com/courses/learn-the-command-line learn the command line]
* [http://upload.itcollege.ee/edmund/ some VMs, ISOs]
* [http://www.tecmint.com/10-useful-free-linux-ebooks-for-newbies-and-administrators/ 10 Free Linux Administration e-books]
* [https://www.youtube.com/playlist?list=PLmbPuZ0NsyGS8ef6zaHd2qYylzsHxL63x Introduction to operating systems (videos)]
* [https://en.wikipedia.org/wiki/Operating_system Operating systems (wikipedia article)]
* [http://enos.itcollege.ee/~edmund/osadmin/materials/ additional OSadmin materials]
* [http://enos.itcollege.ee/~edmund/materials/ additional materials]
* [http://debian-handbook.info/browse/stable/ Debian Administrator's Handbook]
* [https://help.ubuntu.com/lts/serverguide/ Ubuntu Server Guide]
* [https://linuxjourney.com/ Linux Journey]
*'''[http://enos.itcollege.ee/~edmund/materials/links.html Linux and other free software related links]'''
* [http://enos.itcollege.ee/~edmund/materials/news.html Linux news, magazines]

==Finding help==
* https://help.ubuntu.com/ - official documentation
* https://ubuntu-manual.org/

Also some searches:

* https://www.startpage.com/do/search?q=ubuntu+tutorial
* https://www.startpage.com/do/search?q=ubuntu+howto
* https://www.startpage.com/do/search?q=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Also I encourage to use YouTube which is full of videotutorials.

* https://www.youtube.com/results?search_query=ubuntu+tutorial
* https://www.youtube.com/results?search_query=ubuntu+howto
* https://www.youtube.com/results?search_query=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Certainly all the information might not be trustful but you can test at first in virtual machine if you have doubts. Also ask from instant messenger ([[#Instant messaging|also current course has one]]), IRC-channels. If you search https://www.startpage.com/do/search?q=ubuntu+irc+channel then you will find them. You will also find IRC client howtos.

Usually if to search "problem name or error message + operating system name, version" (without quotation marks) then you will find results (e.g.: ''bluetooth ubuntu 16.04''). Usually one of the findings will solve a problem or at least give hints what to do and possible workarounds.

Also using different search engines might give different results. In addition of [https://www.google.com/ Google], you can try also [https://www.startpage.com/ Startpage], [https://duckduckgo.com/ DuckDuckGo] etc - https://en.wikipedia.org/wiki/List_of_search_engines

=Lab works=
Here are some introductory lab tasks available. Labs for assessment are coming into I-Tee virtual laboratory system.

Please check also [https://enos.itcollege.ee/~edmund/osadmin/eng/labs/ log about lab works].

==Lab 0==

Installing Ubuntu Server LTS

'''Introduction to Unix command line''' (cd, ls, cat, full path, relative path etc)

==Lab 1==

'''Managing users''' (adduser, addgroup, passwd, /etc/passwd, /etc/shadow)

1) Create a user sysadmin

2) Add a new group devops and add a the user sysadmin to a previously created group.

3) Divert the user sysadmin's password hash via cowsay to a file called hash.txt.

4) Lock the user sysadmin and be ready to show me the indication of the user being locked.

5) Change the user's current home directory into /home/unknown so that the files will also be moved to the new location.

'''Managing files''' (mkdir, cp, mv, rm, touch, nano, less, chmod, chown, rwx, 644 etc)

1) Create a folder march in root user directory and for every march day a subfolder with a name day1, day2, day3 … day31. (Example: /root/march/day1 or /root/march/day2 etc)

2) Modify the march folder owner so that it will be student and the new group audio.

3) Modify the march folder's and its subfolders so that the user can do anything, group can do ls in the folder and cd into it and others can't do anything with it.

4) Create a hard link called network to a file /etc/network/interfaces

5) Copy /var/log directory into march folder so that the timestamp and user info will be preserved.

'''Processes and environment variables''' (kill, using directing input/output/error: |, <, >, >>; env, PATH, HOME etc)

1) Divert the list with the student user's groups via cowsay into a fail studgroup.txt.

2) Create a environment variable called MYHOME that has the value of the system's HOME environment variable. (Hint: you have tu use variable symbol here!)

3) Send 2 htop's to the background and be ready to present how you send a kill signal to the first htop by job number and term signal to the second htop by a process number.

4) Create an alias called bye that logs you out of the terminal. Make this alias permanent.

5) Execute a programm called espdiff and diver the standardoutput to a file called okay.txt and the standard error to a file called notokay.txt.

'''Managing software''' (installation, updating, deleting, apt and dpkg utils)

==Lab 2==

'''Managing disks by creating partitions''' (fdisk, mkfs, blkid, mount, umount)

'''Managing swap''' (mkswap, swapon, swapoff)

=Practical test=

==2016==

[https://docs.google.com/document/d/1FGZcqmQQDF1l32uPUJ6n8x2Tc4gK8nuxS-C9esgRqaQ/edit?usp=sharing First practical test 10th of May 2016]

[https://docs.google.com/document/d/1ZCqOOMkx0dwP0QXLIK_yk_08a8whfJmQbYR1mAoSh7M/edit Second pracical test 24th of May 2016]

=Exam=

==2016==

[https://docs.google.com/document/d/1ofiylCw9YAS8_S9YHEc8cZOvEfCfMfs2wDoc44eDyCU/edit?usp=sharing Practical exam]

[https://docs.google.com/document/d/1gkEDb1g1em9UGhj9n_LIwnhp17gY85U9aPtMfGk56_8/edit# Topics] of the oral exam in Spring 2016

=Wiki article=

'''[[OSadmin_wiki_article|for further information please look here]]'''

=Former materials=
* [[Operating systems 2016]]

= Students share ==
[https://kobra.io/#/e/-KjbPHJ0sBTp6Q3QLoIx] Bad documentation

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Operating systems

2017-05-08T09:35:28Z

Eocakovs: /* Students share = */

=Lecturer=

[https://wiki.itcollege.ee/index.php/User:Edmund#in_English Edmund Laugasson]

=Intro=
All subject related infotmation will be put up on Wiki page, due to the possibility to have access to the materials even after the subject has concluded. Materials, such as tests, lectures and links to additional materials, will remain available throughout the subject teaching period.

=Aim of this course=

The aim of this course is to introduce the basics of operating systems and IT system life cycle from the viewpoint of the IT system administrator of operating systems. This subject provides hands-on skills needed to complete other field specific subjects in the curriculum.

Lectures give a theoretical background and the labs give hands-on skills on the same topic using Ubuntu Linux Server.

'''This subject is oriented on hands-on practical assignments to compliment the theoretical side of the subject.'''

Learning outcome 1:

A student who has completed the subject is able to perform the most common administrative tasks (user management, software management, disk usage, process management) in at least one of the most popular operating system on a server.

Learning outcome 2:

A student who has completed the subject understands and is able to explain orally the basic concepts of operating systems and its security aspects.

Learning outcome 3:

The student is able to document an operating system's service from an IT systems administrator's viewpoint.

=Portfolio=
Please fulfill [https://goo.gl/forms/BSEkc2NAoFagQ16R2 the following form].

=Deadlines for assignments=

'''03.04.2017''' - Submission of wiki article's topic

'''02.05.2017''' - lab works defended

'''08.05.2017 23:59''' - Submission of wiki article

'''09.05.2017''' - Pre practical test for students, who have done all of their labs

'''16.05.2017''' - Last option to defend lab work

'''23.05.2017''' - Practical test

All dates are inclusive.

=Instant messaging=
Public chat for any subject related questions can be asked at the channel ''#eitc-osadmin:matrix.org'' and client software: [https://riot.im Riot.im].

[https://riot.im/app Launch Riot here in browser].

=Course materials=

* [http://enos.itcollege.ee/~edmund/osadmin/eng/ Course materials can be found here.]
* [https://echo360.e-ope.ee/ess/portal/section/4ef8a128-857f-4d6c-a532-733f394f70dc recordings are available here]

==Links==
* [https://www.codecademy.com/courses/learn-the-command-line learn the command line]
* [http://upload.itcollege.ee/edmund/ some VMs, ISOs]
* [http://www.tecmint.com/10-useful-free-linux-ebooks-for-newbies-and-administrators/ 10 Free Linux Administration e-books]
* [https://www.youtube.com/playlist?list=PLmbPuZ0NsyGS8ef6zaHd2qYylzsHxL63x Introduction to operating systems (videos)]
* [https://en.wikipedia.org/wiki/Operating_system Operating systems (wikipedia article)]
* [http://enos.itcollege.ee/~edmund/osadmin/materials/ additional OSadmin materials]
* [http://enos.itcollege.ee/~edmund/materials/ additional materials]
* [http://debian-handbook.info/browse/stable/ Debian Administrator's Handbook]
* [https://help.ubuntu.com/lts/serverguide/ Ubuntu Server Guide]
* [https://linuxjourney.com/ Linux Journey]
*'''[http://enos.itcollege.ee/~edmund/materials/links.html Linux and other free software related links]'''
* [http://enos.itcollege.ee/~edmund/materials/news.html Linux news, magazines]

==Finding help==
* https://help.ubuntu.com/ - official documentation
* https://ubuntu-manual.org/

Also some searches:

* https://www.startpage.com/do/search?q=ubuntu+tutorial
* https://www.startpage.com/do/search?q=ubuntu+howto
* https://www.startpage.com/do/search?q=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Also I encourage to use YouTube which is full of videotutorials.

* https://www.youtube.com/results?search_query=ubuntu+tutorial
* https://www.youtube.com/results?search_query=ubuntu+howto
* https://www.youtube.com/results?search_query=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Certainly all the information might not be trustful but you can test at first in virtual machine if you have doubts. Also ask from instant messenger ([[#Instant messaging|also current course has one]]), IRC-channels. If you search https://www.startpage.com/do/search?q=ubuntu+irc+channel then you will find them. You will also find IRC client howtos.

Usually if to search "problem name or error message + operating system name, version" (without quotation marks) then you will find results (e.g.: ''bluetooth ubuntu 16.04''). Usually one of the findings will solve a problem or at least give hints what to do and possible workarounds.

Also using different search engines might give different results. In addition of [https://www.google.com/ Google], you can try also [https://www.startpage.com/ Startpage], [https://duckduckgo.com/ DuckDuckGo] etc - https://en.wikipedia.org/wiki/List_of_search_engines

=Lab works=
Here are some introductory lab tasks available. Labs for assessment are coming into I-Tee virtual laboratory system.

Please check also [https://enos.itcollege.ee/~edmund/osadmin/eng/labs/ log about lab works].

==Lab 0==

Installing Ubuntu Server LTS

'''Introduction to Unix command line''' (cd, ls, cat, full path, relative path etc)

==Lab 1==

'''Managing users''' (adduser, addgroup, passwd, /etc/passwd, /etc/shadow)

1) Create a user sysadmin

2) Add a new group devops and add a the user sysadmin to a previously created group.

3) Divert the user sysadmin's password hash via cowsay to a file called hash.txt.

4) Lock the user sysadmin and be ready to show me the indication of the user being locked.

5) Change the user's current home directory into /home/unknown so that the files will also be moved to the new location.

'''Managing files''' (mkdir, cp, mv, rm, touch, nano, less, chmod, chown, rwx, 644 etc)

1) Create a folder march in root user directory and for every march day a subfolder with a name day1, day2, day3 … day31. (Example: /root/march/day1 or /root/march/day2 etc)

2) Modify the march folder owner so that it will be student and the new group audio.

3) Modify the march folder's and its subfolders so that the user can do anything, group can do ls in the folder and cd into it and others can't do anything with it.

4) Create a hard link called network to a file /etc/network/interfaces

5) Copy /var/log directory into march folder so that the timestamp and user info will be preserved.

'''Processes and environment variables''' (kill, using directing input/output/error: |, <, >, >>; env, PATH, HOME etc)

1) Divert the list with the student user's groups via cowsay into a fail studgroup.txt.

2) Create a environment variable called MYHOME that has the value of the system's HOME environment variable. (Hint: you have tu use variable symbol here!)

3) Send 2 htop's to the background and be ready to present how you send a kill signal to the first htop by job number and term signal to the second htop by a process number.

4) Create an alias called bye that logs you out of the terminal. Make this alias permanent.

5) Execute a programm called espdiff and diver the standardoutput to a file called okay.txt and the standard error to a file called notokay.txt.

'''Managing software''' (installation, updating, deleting, apt and dpkg utils)

==Lab 2==

'''Managing disks by creating partitions''' (fdisk, mkfs, blkid, mount, umount)

'''Managing swap''' (mkswap, swapon, swapoff)

=Practical test=

==2016==

[https://docs.google.com/document/d/1FGZcqmQQDF1l32uPUJ6n8x2Tc4gK8nuxS-C9esgRqaQ/edit?usp=sharing First practical test 10th of May 2016]

[https://docs.google.com/document/d/1ZCqOOMkx0dwP0QXLIK_yk_08a8whfJmQbYR1mAoSh7M/edit Second pracical test 24th of May 2016]

=Exam=

==2016==

[https://docs.google.com/document/d/1ofiylCw9YAS8_S9YHEc8cZOvEfCfMfs2wDoc44eDyCU/edit?usp=sharing Practical exam]

[https://docs.google.com/document/d/1gkEDb1g1em9UGhj9n_LIwnhp17gY85U9aPtMfGk56_8/edit# Topics] of the oral exam in Spring 2016

=Wiki article=

'''[[OSadmin_wiki_article|for further information please look here]]'''

=Former materials=
* [[Operating systems 2016]]

= Students share ==
[https://drive.google.com/open?id=0B8doZmdthcFeNkZhMFFXR0lTTjQ] Bad docs
[https://kobra.io/#/e/-KjbPHJ0sBTp6Q3QLoIx] Bad docs 2

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Operating systems

2017-05-08T09:22:28Z

Eocakovs:

=Lecturer=

[https://wiki.itcollege.ee/index.php/User:Edmund#in_English Edmund Laugasson]

=Intro=
All subject related infotmation will be put up on Wiki page, due to the possibility to have access to the materials even after the subject has concluded. Materials, such as tests, lectures and links to additional materials, will remain available throughout the subject teaching period.

=Aim of this course=

The aim of this course is to introduce the basics of operating systems and IT system life cycle from the viewpoint of the IT system administrator of operating systems. This subject provides hands-on skills needed to complete other field specific subjects in the curriculum.

Lectures give a theoretical background and the labs give hands-on skills on the same topic using Ubuntu Linux Server.

'''This subject is oriented on hands-on practical assignments to compliment the theoretical side of the subject.'''

Learning outcome 1:

A student who has completed the subject is able to perform the most common administrative tasks (user management, software management, disk usage, process management) in at least one of the most popular operating system on a server.

Learning outcome 2:

A student who has completed the subject understands and is able to explain orally the basic concepts of operating systems and its security aspects.

Learning outcome 3:

The student is able to document an operating system's service from an IT systems administrator's viewpoint.

=Portfolio=
Please fulfill [https://goo.gl/forms/BSEkc2NAoFagQ16R2 the following form].

=Deadlines for assignments=

'''03.04.2017''' - Submission of wiki article's topic

'''02.05.2017''' - lab works defended

'''08.05.2017 23:59''' - Submission of wiki article

'''09.05.2017''' - Pre practical test for students, who have done all of their labs

'''16.05.2017''' - Last option to defend lab work

'''23.05.2017''' - Practical test

All dates are inclusive.

=Instant messaging=
Public chat for any subject related questions can be asked at the channel ''#eitc-osadmin:matrix.org'' and client software: [https://riot.im Riot.im].

[https://riot.im/app Launch Riot here in browser].

=Course materials=

* [http://enos.itcollege.ee/~edmund/osadmin/eng/ Course materials can be found here.]
* [https://echo360.e-ope.ee/ess/portal/section/4ef8a128-857f-4d6c-a532-733f394f70dc recordings are available here]

==Links==
* [https://www.codecademy.com/courses/learn-the-command-line learn the command line]
* [http://upload.itcollege.ee/edmund/ some VMs, ISOs]
* [http://www.tecmint.com/10-useful-free-linux-ebooks-for-newbies-and-administrators/ 10 Free Linux Administration e-books]
* [https://www.youtube.com/playlist?list=PLmbPuZ0NsyGS8ef6zaHd2qYylzsHxL63x Introduction to operating systems (videos)]
* [https://en.wikipedia.org/wiki/Operating_system Operating systems (wikipedia article)]
* [http://enos.itcollege.ee/~edmund/osadmin/materials/ additional OSadmin materials]
* [http://enos.itcollege.ee/~edmund/materials/ additional materials]
* [http://debian-handbook.info/browse/stable/ Debian Administrator's Handbook]
* [https://help.ubuntu.com/lts/serverguide/ Ubuntu Server Guide]
* [https://linuxjourney.com/ Linux Journey]
*'''[http://enos.itcollege.ee/~edmund/materials/links.html Linux and other free software related links]'''
* [http://enos.itcollege.ee/~edmund/materials/news.html Linux news, magazines]

==Finding help==
* https://help.ubuntu.com/ - official documentation
* https://ubuntu-manual.org/

Also some searches:

* https://www.startpage.com/do/search?q=ubuntu+tutorial
* https://www.startpage.com/do/search?q=ubuntu+howto
* https://www.startpage.com/do/search?q=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Also I encourage to use YouTube which is full of videotutorials.

* https://www.youtube.com/results?search_query=ubuntu+tutorial
* https://www.youtube.com/results?search_query=ubuntu+howto
* https://www.youtube.com/results?search_query=ubuntu+guide

Replace the word "ubuntu" with "linux" and there can be find yet more materials.

Certainly all the information might not be trustful but you can test at first in virtual machine if you have doubts. Also ask from instant messenger ([[#Instant messaging|also current course has one]]), IRC-channels. If you search https://www.startpage.com/do/search?q=ubuntu+irc+channel then you will find them. You will also find IRC client howtos.

Usually if to search "problem name or error message + operating system name, version" (without quotation marks) then you will find results (e.g.: ''bluetooth ubuntu 16.04''). Usually one of the findings will solve a problem or at least give hints what to do and possible workarounds.

Also using different search engines might give different results. In addition of [https://www.google.com/ Google], you can try also [https://www.startpage.com/ Startpage], [https://duckduckgo.com/ DuckDuckGo] etc - https://en.wikipedia.org/wiki/List_of_search_engines

=Lab works=
Here are some introductory lab tasks available. Labs for assessment are coming into I-Tee virtual laboratory system.

Please check also [https://enos.itcollege.ee/~edmund/osadmin/eng/labs/ log about lab works].

==Lab 0==

Installing Ubuntu Server LTS

'''Introduction to Unix command line''' (cd, ls, cat, full path, relative path etc)

==Lab 1==

'''Managing users''' (adduser, addgroup, passwd, /etc/passwd, /etc/shadow)

1) Create a user sysadmin

2) Add a new group devops and add a the user sysadmin to a previously created group.

3) Divert the user sysadmin's password hash via cowsay to a file called hash.txt.

4) Lock the user sysadmin and be ready to show me the indication of the user being locked.

5) Change the user's current home directory into /home/unknown so that the files will also be moved to the new location.

'''Managing files''' (mkdir, cp, mv, rm, touch, nano, less, chmod, chown, rwx, 644 etc)

1) Create a folder march in root user directory and for every march day a subfolder with a name day1, day2, day3 … day31. (Example: /root/march/day1 or /root/march/day2 etc)

2) Modify the march folder owner so that it will be student and the new group audio.

3) Modify the march folder's and its subfolders so that the user can do anything, group can do ls in the folder and cd into it and others can't do anything with it.

4) Create a hard link called network to a file /etc/network/interfaces

5) Copy /var/log directory into march folder so that the timestamp and user info will be preserved.

'''Processes and environment variables''' (kill, using directing input/output/error: |, <, >, >>; env, PATH, HOME etc)

1) Divert the list with the student user's groups via cowsay into a fail studgroup.txt.

2) Create a environment variable called MYHOME that has the value of the system's HOME environment variable. (Hint: you have tu use variable symbol here!)

3) Send 2 htop's to the background and be ready to present how you send a kill signal to the first htop by job number and term signal to the second htop by a process number.

4) Create an alias called bye that logs you out of the terminal. Make this alias permanent.

5) Execute a programm called espdiff and diver the standardoutput to a file called okay.txt and the standard error to a file called notokay.txt.

'''Managing software''' (installation, updating, deleting, apt and dpkg utils)

==Lab 2==

'''Managing disks by creating partitions''' (fdisk, mkfs, blkid, mount, umount)

'''Managing swap''' (mkswap, swapon, swapoff)

=Practical test=

==2016==

[https://docs.google.com/document/d/1FGZcqmQQDF1l32uPUJ6n8x2Tc4gK8nuxS-C9esgRqaQ/edit?usp=sharing First practical test 10th of May 2016]

[https://docs.google.com/document/d/1ZCqOOMkx0dwP0QXLIK_yk_08a8whfJmQbYR1mAoSh7M/edit Second pracical test 24th of May 2016]

=Exam=

==2016==

[https://docs.google.com/document/d/1ofiylCw9YAS8_S9YHEc8cZOvEfCfMfs2wDoc44eDyCU/edit?usp=sharing Practical exam]

[https://docs.google.com/document/d/1gkEDb1g1em9UGhj9n_LIwnhp17gY85U9aPtMfGk56_8/edit# Topics] of the oral exam in Spring 2016

=Wiki article=

'''[[OSadmin_wiki_article|for further information please look here]]'''

=Former materials=
* [[Operating systems 2016]]

= Students share ==
[https://drive.google.com/open?id=0B8doZmdthcFeNkZhMFFXR0lTTjQ] Bad docs

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:55:56Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.

I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

Additionally here are two great articles that will explain SSH more in depth:
[https://wiki.itcollege.ee/index.php/SSH_for_beginners] SSH for beginners by Etienne Barrier.
[https://wiki.itcollege.ee/index.php/SSH_Encryption] SSH Encryption by Frank Korving.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:This will generate an RSA private key <code>~/.ssh/key_file</code> and public key <code>~/.ssh/key_file.pub</code>.

:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:55:33Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.

I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

Additionally here are two great articles that will explain SHH more in depth:
[https://wiki.itcollege.ee/index.php/SSH_for_beginners] SSH for beginners by Etienne Barrier.
[https://wiki.itcollege.ee/index.php/SSH_Encryption] SSH Encryption by Frank Korving.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:This will generate an RSA private key <code>~/.ssh/key_file</code> and public key <code>~/.ssh/key_file.pub</code>.

:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:54:48Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.

I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

Additionally here are two great articles that will explain SHH more in depth:
1. [https://wiki.itcollege.ee/index.php/SSH_for_beginners] SSH for beginners by Etienne Barrier.
2. [https://wiki.itcollege.ee/index.php/SSH_Encryption] SSH Encryption by Frank Korving.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:This will generate an RSA private key <code>~/.ssh/key_file</code> and public key <code>~/.ssh/key_file.pub</code>.

:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:53:36Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.

<pre>I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

Additionally here are two great articles that will explain SHH more in depth:
1. [https://wiki.itcollege.ee/index.php/SSH_for_beginners] SSH for beginners by Etienne Barrier.
2. [https://wiki.itcollege.ee/index.php/SSH_Encryption] SSH Encryption by Frank Korving</pre>

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:This will generate an RSA private key <code>~/.ssh/key_file</code> and public key <code>~/.ssh/key_file.pub</code>.

:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:53:13Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.

<pre>I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
Additionally here are two great articles that will explain SHH more in depth:
1. [https://wiki.itcollege.ee/index.php/SSH_for_beginners] SSH for beginners by Etienne Barrier.
2. [https://wiki.itcollege.ee/index.php/SSH_Encryption] SSH Encryption by Frank Korving</pre>

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:This will generate an RSA private key <code>~/.ssh/key_file</code> and public key <code>~/.ssh/key_file.pub</code>.

:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:46:20Z

Eocakovs: /* Tips and tricks */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:44:10Z

Eocakovs: /* Remote source or destination */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:43:40Z

Eocakovs: /* Remote source or destination */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be <code>~/.ssh/id_rsa</code> if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T07:42:36Z

Eocakovs: /* Remote source or destination */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key' *.c user@10.0.2.20:src/</pre>

<code>/path/to/private_key</code> will usually be ~/.ssh/id_rsa if you generated a key pair using <code>ssh-keygen -f ~/.ssh/id_rsa -t rsa -b 4096</code>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/private_key' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T06:32:24Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -a 1000 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

:Please remeber that file permissions for ~/.ssh directory and its subfiles needs to be strict, to make sure of that run the following commands:
:<pre>chmod 700 ~/.ssh/</pre>
:<pre>chmod 600 ~/.ssh/*</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T06:24:17Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist*</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T06:23:24Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install ssh</pre>
:This will install latest SSH client and server, you can also specify individual pacages, like shown below.
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:*For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-08T06:19:55Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen -t rsa -b 4096 -f ~/.ssh/key_file</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Talk:Pass

2017-05-08T06:12:28Z

Eocakovs: Created page with "Would be so much more beneficial to English study mates if it was.... well in English."

Would be so much more beneficial to English study mates if it was.... well in English.

Rsync eng

2017-05-07T18:51:52Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:51:41Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.

;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:51:14Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;And you have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:44:46Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines. Leave the passphrase empty if you would like to use this authentication method for automated scripts.
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:38:45Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.

:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:37:00Z

Eocakovs: /* Daemon mode */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<pre>sudo su - rsyncuser</pre>
<pre>sudo chmod 600 /path/to/rsyncd.scrt</pre></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T18:34:03Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<code>sudo su - rsyncuser</code>
<code>sudo chmod 600 /path/to/rsyncd.scrt</code></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client openssh-blacklist</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T15:54:17Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<code>sudo su - rsyncuser</code>
<code>sudo chmod 600 /path/to/rsyncd.scrt</code></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to ssh and sshd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openssh-server openssh-client</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/ssh start</pre>
:or
:<pre>sudo service ssh start</pre>
:*Generate keys on both machines
:<pre>ssh-keygen</pre>
:* Copy public key from client to server
:<pre>ssh-copy-id -i ~/.ssh/key_file user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T15:50:51Z

Eocakovs: /* Via remote shell */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<code>sudo su - rsyncuser</code>
<code>sudo chmod 600 /path/to/rsyncd.scrt</code></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to SSH and SSHd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openSSH-server openSSH-client</pre>
:For Fedora and OpenSUSE machines use <code>dnf</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/SSH start</pre>
:or
:<pre>sudo service SSH start</pre>
:*Generate keys on both machines
:<pre>SSH-keygen</pre>
:* Copy public key from client to server
:<pre>SSH-copy-id -i ~/.ssh/rsa_key.pub user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T15:50:38Z

Eocakovs: /* Local-only */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo dnf install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<code>sudo su - rsyncuser</code>
<code>sudo chmod 600 /path/to/rsyncd.scrt</code></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to SSH and SSHd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openSSH-server openSSH-client</pre>
:For Fedora and OpenSUSE machines use <code>yum</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/SSH start</pre>
:or
:<pre>sudo service SSH start</pre>
:*Generate keys on both machines
:<pre>SSH-keygen</pre>
:* Copy public key from client to server
:<pre>SSH-copy-id -i ~/.ssh/rsa_key.pub user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]

Rsync eng

2017-05-07T15:49:46Z

Eocakovs: /* Local-only */

== Summary ==

Rsync is a command line tool used to copy files locally and over the network. To quote the official website: "Rsync - a fast, versatile, remote (and local) file-copying tool" <ref name="rsync man">[https://download.samba.org/pub/rsync/rsync.html] Rsync official man page</ref>. The main draw of Rsync is that it tries to copy only differences between files and not the entire file. Which in turn reduces the data traffic on the network and time spent. This is great, if you ever have tried to make efficient backups over network you will appreciate what authors of Rsync have done. Not only that, Rsync incorporates data compression for even grater time and data transfer efficiency.

But wait, there is more - Rsync has built in ability to copy links, devices, owners, groups and permissions. It can be tunneled via SSH. And supports two way transfer.

In short - you can use Rsync to make backups, mirror file systems or any number of similar operations in a fast and secure way.

How can it transfer only file differences you ask? It has its own algorithm that accomplishes that, section [[Rsync_eng#How it works?|How it works?]] will hopefully provide some answers.

=== Key features <ref name="rsync man" /> ===

* Support for copying links, devices, owners, groups and permissions
* Exclude and exclude-from options similar to GNU tar
* A CVS exclude mode for ignoring the same files that CVS would ignore
* Does not require root privileges
* Pipelining of file transfers to minimize latency costs
* Support for anonymous or authenticated Rsync servers (ideal for mirroring)

== Table of content ==

__TOC__

== How it works? ==

The way Rsync works is that when an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

* Via remote shell

When Rsync communicates with a remote non-daemon server via a remote shell both the Rsync client and server are communicating via pipes through the remote shell. As far as the Rsync processes are concerned there is no network. In this mode the Rsync options for the server process are passed on the command-line that is used to start the remote shell. <ref name="How Rsync Works">[https://rsync.samba.org/how-rsync-works.html] How Rsync Works A Practical Overview</ref>

* Daemon mode

Network socket is used when Rsync is communicating with a daemon. This is the only sort of Rsync communication that could be called network aware. In this mode the Rsync options must be sent over the socket. <ref name="How Rsync Works" />

* Local-only

When Rsync is preforming a local only job the client will fork a server process to become both sender and receiver.

At the very start of the connection client and server agree on communication protocol version by sending their protocol versions to each other and the minimum version value is used. After connection has been established the side that will be sending the files start generating file list. While file list is being generated each entry in the list is sent to the receiving end in compressed format. When file list is generated both sides sort the file list in alphabetical order.

After both sides have the file list sorted the receiving side will run the generator process, it will compare the file list with its local directory tree. During this comparison each file will be checked if it can be skipped, it will never skip directories, device nodes and symlinks. Also missing directories will be created. When generator process determents that a file is not to be skipped that files original version on receiving end will be considered as data source for the transfer and will be used to eliminate the need to transfer already existing data. Elimination process will be made by taking several block checksum and index checksum pairs of the original file (block checksum size and amount is dependent on size of the file). Each checksum pair is then sent to the data sender (sending side).

When sending side receives the checksums of a file it will generate a hash-table index by index checksums of the original file. Then the local file is read and a index checksum is generated for the block beginning with the first byte of the local file. This index checksum is then compared to hash-table that was previously generated, if a match is found then a block checksum is generated of the local file from the same byte, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum” <ref name="How Rsync Works" />.

All the matched an non-matching data is sent to the receiving side. The important part here is that for matched data only block and index checksums are sent, with requires very little network utilization, only for non-matching data checksums and data is sent. Non-matching data will be sent to the receiver followed by the offset and length in the original file of the matching block and the block checksum generator will be advanced to the next byte after the matching block. Matching blocks can be identified in this way even if the blocks are reordered or at different offsets <ref name="How Rsync Works" />.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfer), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file. Generating the rolling checksums and searching for matches in the checksum set sent by the receiving side require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The receiver will read from the sender data for each file identified by the index checksum. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance <ref name="How Rsync Works" />.

If you are interested how well Rsync algorithm preforms I suggest you read [http://www-2.cs.cmu.edu/~jcl/research/mrsync/mrsync.ps Multiround Rsync] by John Langford. He compares Rsync algorithm to his own improved version and by doing that has provided quite detailed analysis of Rsync algorithm performance.

== Quick start guide ==

For people who are in a hurry.

;Debian based machine and Rsync in local-only mode'''

Open terminal and paste the following

<pre>sudo apt install rsync
rsync -av ~/ /tmp/my_local_backup</pre>
Congratulations! you have made a backup of your home directory.

== Setup ==

;Setup will not cover Windows machines''' If you are interested in setting up Rsync vis SSH on Windows I suggest looking at [https://itefix.net/free itefix] solutions for installing SSH and Rsync.

As mentioned in the beginning of [[Rsync_eng#How it works?|How it works?]] section there are several ways Rsync can be used - in a daemon mode, via remote shell or locally.

Each of these cases will require slightly different setup process. Because of that reason I have split up setup in 3 subsections for each use case.

=== Local-only ===

In this case you will need only Rsync itself and all the transfer parameters will be passed to Rsync itself as command line options.

First check if you have Rsync already installed by running:

<pre>which rsync</pre>
If the the output returns a path to rsync then you are all done and can skip directly to [[Rsync_eng#Usage|Usage]] section.

Otherwise depending on your system run the following commands:

* Debian based machine
<pre>sudo apt install rsync</pre>
* Fedora machine
<pre>sudo yum install rsync</pre>
* OpenSUSE
<pre>sudo zypper install rsync</pre>

=== Daemon mode ===

In this case, the way Rsync will work is that at least one of the machines involved in data transfer needs to be an "rsync server" by running Rsync in a daemon mode (<code>rsync --daemon</code> at the command line) and setting up a short, easy configuration file (<code>/etc/rsyncd.conf</code>) <ref name="Rsync tutorial">[http://everythinglinux.org/rsync/] Tutorial on using Rsync</ref>.

Any number of machines with Rsync installed may then synchronize to and/or from the machine running the Rsync daemon. <ref name="Rsync tutorial" />

This way of using Rsync is beneficial if you don't want or need the client to specify the file transfer options and path, since you can explicitly specify what and how can be transfered to or from remote machine.

First follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on '''all''' machines involved in the transfer process.

When Rsync is installed you need to set up Rsync in a daemon mode on at least one of the machines involved in data transfer. For that we will set up a configuration file on that machine.

Open <code>/etc/rsyncd.conf</code> in your favorite text editor and paste the following:

<source lang="ini">motd file = /path/to/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = local5
reverse lookup = no

[no_security_module]
path = /path/to/files ./pub
comment = Some files to sync (approx 6.1 GB)

[more_security_module]
path = /path/to/files
comment = Some files to sync (approx 10.2 GB)
uid = nobody
gid = nobody
read only = no
list = yes
auth users = username, anotheruser
secrets file = /path/to/rsyncd.scrt
reverse lookup = yes
hosts allow = 10.0.1.12, 192.168.0.0/16, fe80::/64, 172.16.143.*
refuse options = c delete
max connections = 4</source>

;To see detailed explanation of the parameters we pasted to <code>/etc/rsyncd.conf</code> see [[Rsync_eng#rsyncd.conf parameters|rsyncd.conf parameters]] section.'''

Parameters shown above can be adjured to your personal preference. I have shown 2 different modules, that are marked by square brackets surrounding them. 'no_security_module' module is a very basic one that just mentions a path to be synced and a comment. 'more_security_module' one is more complex and is aimed at securing the connection if secure remote shell is not used.

If you will be using the more secure module then open <code>/path/to/rsyncd.scrt</code> in your favorite text editor and add user names and passwords for users you wrote in <code>auth users</code> parameter. Format for the file is as follows:

<pre>username:password
bob:bobspass
sally:herpass</pre>
Please note the following:

<ul>
<li>Users that you list in <code>/etc/rsyncd.conf</code> and <code>/path/to/rsyncd.scrt</code> are not your system users, they are arbitrary users that will be used only for Rsync process.</li>
<li>All the paths listed in <code>/etc/rsyncd.conf</code> need to be accessible by Rsync.</li>
<li>It is important that <code>rsyncd.scrt</code> file must be accessible only to user that runs Rsync daemon, because it contains user name and password information. I would suggest using the following commands:
<code>sudo su - rsyncuser</code>
<code>sudo chmod 600 /path/to/rsyncd.scrt</code></li>
<li>Rsync daemon usually listens to TCP port 873, so don't forget to white-list it in you firewall.</li></ul>

After configuration file has been made you can start up Rsync in daemon mode by running <code>rsync --daemon</code>.

Later on if you want the daemon process to be started automatically you can either use inet daemon or place <code>rsync --daemon</code> command in a shell script and add it to startup, both methods have their merit, which one you use is up to you.

=== Via remote shell ===

One of the most convenient ways to use Rsync is via remote shell, it will allow you to bypass setting up Rsync built in authentication system. Also It will provide security layer since Rsync authentication protocol is a 128 bit MD4 based challenge response system <ref name="Rsync daemon man">[https://download.samba.org/pub/rsync/rsyncd.conf.html] Rsync daemon configuration file man page</ref> which is quite poor. On top of that using SSH will provide data encryption.

That is why I suggest using [https://kimmo.suominen.com/docs/SSH/ SSH] as your remote shell with Rsync.

Before dealing with SSH follow the steps listed in [[Rsync_eng#Setup|Setup]] Local-only subsection on all machines involved in the transfer process. When that is done, follow the steps below.

;First you need to makes sure you have SSH client installed.
:On Unix like machine you could do the following:
:<pre>which ssh</pre>
;You have SSH server installed and running on the server machine.
:You can do it with the following command:
:<pre>which sshd</pre>
:If the the output shows path to SSH and SSHd then you can skip this step.
:Otherwise use following commands.
:* Debian based machine
:<pre>sudo apt install openSSH-server openSSH-client</pre>
:For Fedora and OpenSUSE machines use <code>yum</code> and <code>zypper</code> respectively.
:
;At this point you should have both Rsync and SSH on all machines involved in the data transfer.
:I will not go trough full SSH setup, for that please see [http://troy.jdmz.net/rsync/index.html this] link.
:The basics go like this:
:* Start up sshd process on server
:<pre>sudo /etc/init.d/SSH start</pre>
:or
:<pre>sudo service SSH start</pre>
:*Generate keys on both machines
:<pre>SSH-keygen</pre>
:* Copy public key from client to server
:<pre>SSH-copy-id -i ~/.ssh/rsa_key.pub user@IP-address</pre>

When Rsync is used via SSH you can still use Rsync in daemon mode to use preconfigure modules, see subsection Daemon mode of [[Rsync_eng#Setup|Setup]]. In that case only the usage syntax will change as specified in [[Rsync_eng#Usage|Usage]] section.

== Synopsis ==

<source lang="ini">Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
</source>

Usages with just one SRC arg and no DEST arg will list the source files instead of copying <ref name="rsync man" />.

== Usage ==

You use Rsync in the same way you use <code>rcp</code>. You must specify a source and a destination, one of which may be remote <ref name="rsync man" />.

All the Rsync command line options used below are explained in [[Rsync_eng#Command options|Command options]] section.

=== Local-only ===

Rsync can be used to copy files to remote detestation or locally. In case of local-only use it behaves like an improved copy command.

Command sample:

<pre>rsync -t *.c src/</pre>
This would transfer all files matching the pattern *.c from the current directory to the directory src. If any of the files already exist on the remote system then the Rsync remote-update protocol is used to update the file by sending only the differences <ref name="rsync man" />.

Another way is to specify the directory you would like to copy. It can be done in two ways. One is by including trailing slash in the source path, like so:

<pre>rsync -av /path/to/source/ /path/to/destination</pre>
This will copy all the content of <code>source</code> directory to <code>destination</code> directory.

Second option is to omit the trailing slash, like so:

<pre>rsync -av /path/to/source /path/to/destination</pre>
This would will copy all the content of <code>source</code> directory, including the directory itself to <code>destination</code> directory, so in the end we will have the content of <code>source</code> in this path - <code>/path/to/destination/source</code>.

To copy directories or files that include white spaces surround them with <code>'</code> or in newer versions of Rsync use the '''--protect-args (-s)''' option.

<pre>rsync '/file with spaces' /path/to/destination

rsync -s /file with spaces /path/to/destination</pre>
If you would like to transfer several files from the source to the same destination you would do something like this:

<pre>rsync -av /file1 /file2 /path/to/destination</pre>

=== Remote source or destination ===

When remote source or destination is used a way of contacting remote system must be specified. There are two ways:

* Using a remote-shell program as the transport (such as SSH). When using this method source or destination path contains a single colon (:) separator after a host specification.
* Contacting an Rsync daemon directly via TCP This happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified

There is however exception to this rule, if double colon (::) separator syntax is used and remote shell is specified via --rsh (-e) option then a single use daemon will be spawned on remote host that will read its configuration file from specified users home directory. This however is not the best way to secure a daemon transfer. Better way would be using ssh to tunnel a local port to a remote machine and configure a normal rsync daemon on that remote host to only allow connections from "localhost" <ref name="rsync man" />.

For instructions on SSH port forwarding see [https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding this].

Local source to remote destination command sample:

<pre>rsync -t *.c foo:src/</pre>
This the same as local-only sample only it copies files to the directory src on the machine foo.

To expand on previous sample depending on the remote host setup.

* Daemon mode

<pre>rsync -tavz *.c username@10.0.2.20::module_name</pre>

* Remote shell

<pre>rsync -tavz -e 'ssh -i /path/to/private_key.pem' *.c user@10.0.2.20:src/</pre>

As you can see in daemon mode we specify remote destination ip followed by <code>::</code> and the module we want to use, as per installation instructions module contains all the configuration options. And note that <code>username</code> is a user specified in <code>/path/to/rsyncd.scrt</code>.

In remote shell mode we specify remote shell via -e option and SSH <code>user</code> followed by <code>@</code> and ip of the destination followed by <code>:</code> and destination path.

A bit more complex sample using ssh from remote source to local destination.

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/file{1,2} :/file3 user@172.16.1.10:/file4 /path/to/destination</pre>
This would transfer files - file1, file2, file3 from <code>10.0.2.20</code> and file4 from <code>172.16.1.10</code> to /path/to/destination on local machine. It shows the versatility of Rsync, you can specify several individual files on remote host - <code>:/file1 :/file2</code> or you can specify a pattern <code>/file{1,2}</code>. Also you can specify several remote hosts if the ssh key you provided will match public key on remote machines.

You can do similarly if transferring in daemon mode:

<pre>rsync -avz user@10.0.2.20::module_name/file{1,2} user@172.16.1.10::module_name/file3 /path/to/destination</pre>
Directory copy works the same as in local-only mode only difference is that you have to specify remote host:

<pre>rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</pre>

or

<pre>rsync -avz user@10.0.2.20::module_name /path/to/destination</pre>
One important thing to note that host and module references don't require a trailing slash to copy the contents of the default directory <ref name="rsync man" />.

=== Special case ===

If a single source argument is specified without a destination, the files are listed in an output format similar to <code>ls -l </code> <ref name="rsync man" />.

=== Tips and tricks ===

;Use filters to exclude or include files.
: This is useful if for example you would like to transfer specific pattern and exclude some fies from that pattern or exclude all files but the ones you specify in include rule.
:There are several ways of doing it:
:* filter option
:<pre>rsync -av --filter '- *.tmp' /path/to/source/ /path/to/destination</pre>
:This would exclude files with <code>*.tmp</code> pattern.
:
:* exclude / include options
:<pre>rsync -av --exclude '*.tmp' /path/to/source/ /path/to/destination</pre>
:This would do the same as above sample.
:<pre>rsync -av --include '*.txt' /path/to/source/ /path/to/destination</pre>
:This would include <code>*.txt</code> file pattern, it would be useful only in combination with exclude pattern, because Rsync will transfer all the files anyway if exclude option is not specified.
:
:*Using exclude / include file
:It is a bit more versatile method, because you don't have to clutter your command line options with possibly dozens of filters.
:To pass file with filters you would do something like so:
:<pre>rsync -av --exclude-from '/exclude_file' --include-from '/include_file' /path/to/source/ /path/to/destination</pre>
:And the files themselves would have the new line separated exclude / include pattern:
:<pre>*.temp /some/dir *.txt */some/other/dir</pre>
:Also not that if you use <code>*</code> as exclude rule everything will be excluded, so if you want to exclude everything except specific pattern first specify an include rule something like <code>*/</code> that would tell to include the directory itself

;Store a password file on local machine to avoid typing password when transferring daemon mode.
:Some modules on the remote daemon may require authentication. If so, you will receive a password prompt when you connect. You can avoid the password prompt by setting the environment variable RSYNC_PASSWORD to the password you want to use or using the --password-file option. This may be useful when scripting rsync.
:WARNING: On some systems environment variables are visible to all users. On those systems using --password-file is recommended </code> <ref name="rsync man" />.

;Set up scripts or make files together with cron jobs to automate the process.
:First you would need to create a script on make file, you can do that for example by opening a file in text editor:
:<pre>nano ~/backup_files.sh</pre>
:or
:<pre>nano ~/Makefile</pre>
:Depending witch option you chose you would write something like this in to script file:
<source lang="bash">#!/bin/bash
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:And something like this in to Makefile:
<source lang="make">backup:
rsync -avz -e 'ssh -i /path/to/pemfile.pem' user@10.0.2.20:/path/to/source /path/to/destination</source>
:Then you would edit your Crontab like so:
:<pre>crontab -e</pre>
:And there you would write something like this for Bash script:
:<pre>5 0 * * * $HOME/backup_files.sh</pre>

You can find out about Bash, Makefile and Crontab below.

[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_01.html Bash]
[http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ Makefile]
[https://linux.die.net/man/1/crontab Crontab]

== Command options ==

Rsync options used in this article can be found below.

This section is filtered copy from man <ref name="rsync man" />

Rsync accepts both long (double-dash + word) and short (single-dash + letter) options.

;-a, --archive
:This is equivalent to '''-rlptgoD'''. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when '''--files-from''' is specified, in which case -r is not implied. Note that '''-a does not preserve hardlinks''', because finding multiply-linked files is expensive. You must separately specify -H.

;-v, --verbose
:This option increases the amount of information you are given during the transfer. By default, rsync works silently. A single '''-v''' will give you information about what files are being transferred and a brief summary at the end. Two '''-v''' options will give you information on what files are being skipped and slightly more information at the end. More than two '''-v''' options should only be used if you are debugging rsync. In a modern rsync, the '''-v''' option is equivalent to the setting of groups of '''--info''' and '''--debug''' options. You can choose to use these newer options in addition to, or in place of using '''--verbose''', as any fine-grained settings override the implied settings of '''-v'''. Both '''--info''' and '''--debug''' have a way to ask for help that tells you exactly what flags are set for each increase in verbosity.

:However, do keep in mind that a daemon's "max verbosity" setting will limit how high of a level the various individual flags can be set on the daemon side. For instance, if the max is 2, then any info and/or debug flag that is set to a higher value than what would be set by '''-vv''' will be downgraded to the '''-vv''' level in the daemon's logging.

;-z, --compress
:With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. This matching-data compression comes at a cost of CPU, though, and can be disabled by repeating the -z option, but only if both sides are at least version 3.1.1.

:Note that if your version of rsync was compiled with an external zlib (instead of the zlib that comes packaged with rsync) then it will not support the old-style compression, only the new-style (repeated-option) compression. In the future this new-style compression will likely become the default.

:The client rsync requests new-style compression on the server via the '''--new-compress''' option, so if you see that option rejected it means that the server is not new enough to support '''-zz'''. Rsync also accepts the '''--old-compress''' option for a future time when new-style compression becomes the default.

:See the '''--skip-compress''' option for the default list of file suffixes that will not be compressed.

;-t, --times
:This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing '''-t''' or '''-a''' will cause the next transfer to behave as if it used '''-I''', causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using '''-t''').

;-e, --rsh=COMMAND
:This option allows you to choose an alternative remote shell program to use for communication between the local and remote copies of rsync. Typically, rsync is configured to use ssh by default, but you may prefer to use rsh on a local network. If this option is used with '''[user@]host::module/path''', then the remote shell COMMAND will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync daemon on the remote host. See the section "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

:Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes rsync is parsing). Some examples:

:<pre>-e 'ssh -p 2234' -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'</pre>

:(Note that ssh users can alternately customize site-specific connect options in their .ssh/config file.)

:You can also choose the remote shell program using the RSYNC_RSH environment variable, which accepts the same range of values as -e.

:See also the '''--blocking-io''' option which is affected by this option.

;-f, --filter=RULE
:This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer. You may use as many '''--filter''' options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.

:See the FILTER RULES section for detailed information on this option.

;--exclude=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--exclude-from=FILE
:This option is related to the '''--exclude''' option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

;--include=PATTERN
:This option is a simplified form of the '''--filter''' option that defaults to an include rule and does not allow the full rule-parsing syntax of normal filter rules. See the FILTER RULES section for detailed information on this option.

;--include-from=FILE
:This option is related to the '''--include''' option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input.

All of the options can be found on Rsync [https://download.samba.org/pub/rsync/rsync.html man page]

== rsyncd.conf parameters ==

Rsyncd.conf parameters used in this article can be found below.

This scetion is filtered copy of Rsync daemon man <ref name="Rsync daemon man" />

Configuration file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The first parameters in the file (before a [module] header) are the global parameters. [https://download.samba.org/pub/rsync/rsyncd.conf.html ref]

A useful option is to use references to environment variables in the values of parameters. Like so <code>uid = %RSYNC_USER_NAME%</code> where RSYNC_USER_NAME is an environment variable.

;motd file
:This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the '''--dparam=motdfile=FILE''' command-line option when starting the daemon.

;log file
:When the "log file" parameter is set to a non-empty string, the rsync daemon will log messages to the indicated file rather than using syslog. This is particularly useful on systems (such as AIX) where syslog() doesn't work for chrooted programs. The file is opened before chroot() is called, allowing it to be placed outside the transfer. If this value is set on a per-module basis instead of globally, the global log will still contain any authorization failures or config-file error messages. If the daemon fails to open the specified file, it will fall back to using syslog and output an error about the failure. (Note that the failure to open the specified log file used to be a fatal error.)

:This setting can be overridden by using the '''--log-file=FILE''' or '''--dparam=logfile=FILE''' command-line options. The former overrides all the log-file parameters of the daemon and all module settings. The latter sets the daemon's log file and the default for all the modules, which still allows modules to override the default setting.

;pid file
:This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the '''--dparam=pidfile=FILE''' command-line option when starting the daemon.

;lock file
:This parameter specifies the file to use to support the "max connections" parameter. The rsync daemon uses record locking on this file to ensure that the max connections limit is not exceeded for the modules sharing the lock file. The default is <code>/var/run/rsyncd.lock</code>.

;syslog facility
:This parameter allows you to specify the syslog facility name to use when logging messages from the rsync daemon. You may use any standard syslog facility name which is defined on your system. Common names are auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6 and local7. The default is daemon. This setting has no effect if the "log file" setting is a non-empty string (either set in the per-modules settings, or inherited from the global settings).

;reverse lookup
:Controls whether the daemon performs a reverse lookup on the client's IP address to determine its hostname, which is used for "hosts allow"/"hosts deny" checks and the "%h" log escape. This is enabled by default, but you may wish to disable it to save time if you know the lookup will not return a useful result, in which case the daemon will use the name "UNDETERMINED" instead. If this parameter is enabled globally (even by default), rsync performs the lookup as soon as a client connects, so disabling it for a module will not avoid the lookup. Thus, you probably want to disable it globally and then enable it for modules that need the information.

;path
:This parameter specifies the directory in the daemon's filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path's value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user's name in the path:

:<pre> path = /home/%RSYNC_USER_NAME%</pre>
:It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn't try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace.

;comment
:This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment.

;uid
:This parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the "gid" parameter this determines what file permissions are available. The default when run by a super-user is to switch to the system's "nobody" user. The default for a non-super-user is to not try to change the user. See also the "gid" parameter. The RSYNC_USER_NAME environment variable may be used to request that rsync run as the authorizing user. For example, if you want a rsync to run as the same user that was received for the rsync authentication, this setup is useful:

:<pre>uid = %RSYNC_USER_NAME%</pre>
:<pre>gid = *</pre>
;gid
:This parameter specifies one or more group names/IDs that will be used when accessing the module. The first one will be the default group, and any extra ones be set as supplemental groups. You may also specify a "*" as the first gid in the list, which will be replaced by all the normal groups for the transfer's user (see "uid"). The default when run by a super-user is to switch to your OS's "nobody" (or perhaps "nogroup") group with no other supplementary groups. The default for a non-super-user is to not change any group attributes (and indeed, your OS may not allow a non-super-user to try to change their group settings).

;read only
:This parameter determines whether clients will be able to upload files or not. If "read only" is true then any attempted uploads will fail. If "read only" is false then uploads will be possible if file permissions on the daemon side allow them. The default is for all modules to be read only. Note that "auth users" can override this setting on a per-user basis.

;list
:This parameter determines whether this module is listed when the client asks for a listing of available modules. In addition, if this is false, the daemon will pretend the module does not exist when a client denied by "hosts allow" or "hosts deny" attempts to access it. Realize that if "reverse lookup" is disabled globally but enabled for the module, the resulting reverse lookup to a potentially client-controlled DNS server may still reveal to the client that it hit an existing module. The default is for modules to be listable.

;auth users
:This parameter specifies a comma and/or space-separated list of authorization rules. In its simplest form, you list the usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The rules may contain shell wildcard characters that will be matched against the username provided by the client for authentication. If "auth users" is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication protocol is used for this exchange. The plain text usernames and passwords are stored in the file specified by the "secrets file" parameter. The default is for all users to be able to connect without a password (this is called "anonymous rsync"). In addition to username matching, you can specify groupname matching via a '@' prefix. When using groupname matching, the authenticating username must be a real user on the system, or it will be assumed to be a member of no groups. For example, specifying "@rsync" will match the authenticating user if the named user is a member of the rsync group.

:Finally, options may be specified after a colon (:). The options allow you to "deny" a user or a group, set the access to "ro" (read-only), or set the access to "rw" (read/write). Setting an auth-rule-specific ro/rw setting overrides the module's "read only" setting.

:Be sure to put the rules in the order you want them to be matched, because the checking stops at the first matching user or group, and that is the only auth that is checked. For example:

:<pre>auth users = joe:deny @guest:deny admin:rw @rsync:ro susan joe sam</pre>
:In the above rule, user joe will be denied access no matter what. Any user that is in the group "guest" is also denied access. The user "admin" gets access in read/write mode, but only if the admin user is not in group "guest" (because the admin user-matching rule would never be reached if the user is in group "guest"). Any other user who is in group "rsync" will get read-only access. Finally, users susan, joe, and sam get the ro/rw setting of the module, but only if the user didn't match an earlier group-matching rule.

:If you need to specify a user or group name with a space in it, start your list with a comma to indicate that the list should only be split on commas (though leading and trailing whitespace will also be removed, and empty entries are just ignored). For example:

:<pre>auth users = , joe:deny, @Some Group:deny, admin:rw, @RO Group:ro</pre>
:See the description of the secrets file for how you can have per-user passwords as well as per-group passwords. It also explains how a user can authenticate using their user password or (when applicable) a group password, depending on what rule is being authenticated.

:See also the section entitled "USING RSYNC-DAEMON FEATURES VIA A REMOTE SHELL CONNECTION" in [https://download.samba.org/pub/rsync/rsyncd.conf.html rsync] for information on how handle an rsyncd.conf-level username that differs from the remote-shell-level username when using a remote shell to connect to an rsync daemon.

;secrets file
:This parameter specifies the name of a file that contains the username:password and/or @groupname:password pairs used for authenticating this module. This file is only consulted if the "auth users" parameter is specified. The file is line-based and contains one name:password pair per line. Any line has a hash (#) as the very first character on the line is considered a comment and is skipped. The passwords can contain any characters but be warned that many operating systems limit the length of passwords that can be typed at the client end, so you may find that passwords longer than 8 characters don't work. The use of group-specific lines are only relevant when the module is being authorized using a matching "@groupname" rule. When that happens, the user can be authorized via either their "username:password" line or the "@groupname:password" line for the group that triggered the authentication.

:It is up to you what kind of password entries you want to include, either users, groups, or both. The use of group rules in "auth users" does not require that you specify a group password if you do not want to use shared passwords.

:There is no default for the "secrets file" parameter, you must choose a name (such as <code>/etc/rsyncd.secrets</code>). The file must normally not be readable by "other"; see "strict modes". If the file is not found or is rejected, no logins for a "user auth" module will be possible.

;hosts allow
:This parameter allows you to specify a list of comma- and/or whitespace-separated patterns that are matched against a connecting client's hostname and IP address. If none of the patterns match, then the connection is rejected. Each pattern can be in one of five forms:

:* a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address of the form a:b:c::d:e:f. In this case the incoming machine's IP address must match exactly.
:* an address/mask in the form ipaddr/n where ipaddr is the IP address and n is the number of one bits in the netmask. All IP addresses which match the masked IP address will be allowed in.
:* an address/mask in the form ipaddr/maskaddr where ipaddr is the IP address and maskaddr is the netmask in dotted decimal notation for IPv4, or similar for IPv6, e.g. ffff:ffff:ffff:ffff:: instead of /64. All IP addresses which match the masked IP address will be allowed in.
:* a hostname pattern using wildcards. If the hostname of the connecting IP (as determined by a reverse lookup) matches the wildcarded name (using the same rules as normal unix filename matching), the client is allowed in. This only works if "reverse lookup" is enabled (the default).
:* a hostname. A plain hostname is matched against the reverse DNS of the connecting IP (if "reverse lookup" is enabled), and/or the IP of the given hostname is matched against the connecting IP (if "forward lookup" is enabled, as it is by default). Any match will be allowed in.

:Note IPv6 link-local addresses can have a scope in the address specification:

:<pre>fe80::1%link1</pre>
:<pre>fe80::%link1/64</pre>
:<pre>fe80::%link1/ffff:ffff:ffff:ffff::</pre>
:You can also combine "hosts allow" with a separate "hosts deny" parameter. If both parameters are specified then the "hosts allow" parameter is checked first and a match results in the client being able to connect. The "hosts deny" parameter is then checked and a match means that the host is rejected. If the host does not match either the "hosts allow" or the "hosts deny" patterns then it is allowed to connect.

:The default is no "hosts allow" parameter, which means all hosts can connect.

;refuse options
:This parameter allows you to specify a space-separated list of rsync command line options that will be refused by your rsync daemon. You may specify the full option name, its one-letter abbreviation, or a wild-card string that matches multiple options. For example, this would refuse '''--checksum (-c)''' and all the various delete options:

:<pre> refuse options = c delete</pre>
:The reason the above refuses all delete options is that the options imply '''--delete''', and implied options are refused just like explicit options. As an additional safety feature, the refusal of "delete" also refuses '''remove-source-files''' when the daemon is the sender; if you want the latter without the former, instead refuse "delete-'''" -- that refuses all the delete modes without affecting '''--remove-source-files*.

:When an option is refused, the daemon prints an error message and exits. To prevent all compression when serving files, you can use "dont compress = *" (see below) instead of "refuse options = compress" to avoid returning an error to a client that requests compression.

;max connections
:This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. The default is 0, which means no limit. A negative value disables the module. See also the "lock file" parameter.

= Author =

;Eriks Ocakovskis C11, Estonian IT College, 07-05-2017

= References =

{{Reflist|2}}

[[Category:Operatsioonisüsteemide administreerimine ja sidumine]]