Sync Files Between Linux Servers with Ansible

Keeping files synchronized between Linux servers is a common task. You may want to copy application configuration, NGINX settings, SSL certificates, or other files from one server to another to make both look identical.

Ansible provides several ways to copy files, but for server-to-server synchronization, ansible.posix.synchronize is usually the best choice. It uses rsync, so only changed files are transferred, making it much faster than copying everything every time.

The playbook performs four steps:

  • Check whether the target server already has an SSH key.
  • Upload the SSH key if it is missing.
  • Allow the target server to connect to the source server.
  • Synchronize the required directories or files.

Prerequisites: network connection between servers, configured sudoers , generated a pair of ssh keys, defined variables used in the playbook

Step 1 – Check whether an SSH key already exists

Before uploading a key, the playbook checks whether one is already present.

- name: Check if id_rsa key already exists (target)
ansible.builtin.stat:
path: ~/.ssh/id_rsa
register: id_rsa
become: true
become_user: "{{ target_host_user }}"
when: ('target_host' in inventory_hostname)

The important parts are:

  • register: id_rsa stores the result of the stat module so later tasks can check whether the file exists.
  • become_user: "{{ target_host_user }}" runs the task as the application user instead of root. This checks the correct home directory.
  • when ensures this task runs only on the target server.

Step 2 – Upload the SSH key

If the key does not exist, create the .ssh directory and upload both the private and public keys.

- name: Set target ssh
when: ('target_host' in inventory_hostname) and (not id_rsa.stat.exists)
become: true
become_user: "{{ target_host_user }}"
block:
- name: Set target ssh | Create directories for ssh (target)
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: "{{ target_host_user }}"
group: "{{ target_host_user }}"
mode: "0700"
loop:
- ~/.ssh
- name: Set target ssh | Upload keys to ssh dir
no_log: true
ansible.builtin.copy:
dest: "~/.ssh/{{ item.file }}"
content: "{{ item.content }}"
mode: "{{ item.mode }}"
loop:
- { file: "id_rsa", content: "{{ target_host_private_key }}", mode: "0600" }
- { file: "id_rsa.pub", content: "{{ target_host_public_key }}", mode: "0644" }

The interesting options are:

  • block groups related tasks under a single when condition.
  • no_log: true prevents the private SSH key from appearing in the Ansible output or logs.
  • loop allows us to interact with multiple files in just a one task
  • The task only runs when id_rsa.stat.exists is false, making the playbook idempotent.

Step 3 – Allow the target server to connect

The source server must trust the target server’s SSH key.

- name: Add target public key to source host
ansible.posix.authorized_key:
user: "{{ source_host_user }}"
key: "{{ target_host_ssh_public_key }}"
state: present
register: auth_key
when: ('source_host' in inventory_hostname)

The authorized_key module adds the target server’s public key to the source server’s authorized_keys file. After this, the target server can connect over SSH without a password.

Step 4 – Synchronize the files

Finally, synchronize the required directories.

- name: Synchronize source and target files (target)
ansible.posix.synchronize:
src: "{{ source_host_user }}@{{ source_host }}:{{ item.src }}"
dest: "{{ item.dest }}"
mode: pull
rsync_path: sudo rsync
rsync_opts:
- "-e 'ssh -i /home/{{ target_host_user }}/.ssh/id_rsa'"
- "--mkpath"
- "--delete"
loop:
- { src: "/etc/nginx/", dest: "/etc/nginx/" }
- { src: "{{ app_dir }}/config/", dest: "{{ app_dir }}/config/" }
- { src: "/home/{{ app_user }}/.somefile", dest: "/home/{{ app_user }}/" }
delegate_to: "{{ inventory_hostname }}"
register: rsync_result
failed_when: rsync_result.rc not in [0, 23]
become: true
when: ('target_host' in inventory_hostname) and (auth_key is defined)

A few options are worth explaining:

  • mode: pull means the target server connects to the source server and downloads the files.
  • src: "{{ source_host_user }}@{{ source_host }}:{{ item.src }}" tells rsync to read the files directly from the remote server over SSH.
  • rsync_path: sudo rsync runs rsync with sudo on the source server, allowing access to protected directories such as /etc/nginx.
  • -e 'ssh -i ...' tells rsync which SSH private key to use.
  • --mkpath creates the destination directory if it does not already exist.
  • --delete removes files that no longer exist on the source server, keeping both servers synchronized.
  • delegate_to: "{{ inventory_hostname }}" makes sure the synchronization runs on the target server. Since we use mode: pull, the target server is responsible for opening the SSH connection.
  • failed_when: rsync_result.rc not in [0, 23] ignores rsync exit code 23, which commonly indicates that a source file or directory does not exist. Depending on your environment, this may be expected and should not fail the playbook.

Push vs Pull

The synchronize module supports two synchronization modes.

  • Push (mode: push) – the source server connects to the destination server and sends the files
  • Pull (mode: pull) – the destination server connects to the source server and downloads the files.

This example uses pull mode because the target server initiates the SSH connection. This is often preferred when the source server should not connect to other machines.

Other ways to copy files

The synchronize module is not the only option.

  • copy – Copies files from the Ansible control node to managed hosts. Best for small files or configuration files.
  • template – Similar to copy, but processes Jinja2 templates before copying.
  • fetch – Copies files from managed hosts back to the Ansible control node.
  • command or shell – Runs rsync directly if you need options that are not supported by the synchronize module.

I hope the post was useful.

Cheers.

Why GitLab Fails with “Operation Not Permitted” on Windows Using Podman

If you run GitLab (or any application that modifies file permissions or ownership of files in volume mounts) in a container, you may see the installation fail with an error like:

chgrp: changing group of '/var/opt/gitlab/git-data/repositories': Operation not permitted

This error prevents GitLab from starting. Here’s why it happens—and the simplest way to fix it.

A local GitLab installation was required to troubleshoot and verify several production-critical queries. This setup is clearly not intended for production use and should be used only for testing and troubleshooting purposes.

Also, Windows is not officially supported as the images have known compatibility issues with volume permissions and potentially other unknown issues (although, I haven’t noticed any issues during a week)

Both Podman Desktop and Docker Desktop run containers by using WSL2

The problem appears when you bind-mount a Windows directory (NTFS) into the container, for example:

E:\volumes\gitlab\data → /var/opt/gitlab
podman run --detach --hostname gitlab.example.com `
--env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.example.com'" `
--publish 443:443 --publish 80:80 --publish 22:22 ` 
--name gitlab --restart always `  
--volume /e/volumes/gitlab/config:/etc/gitlab `
--volume /e/volumes/gitlab/logs:/var/log/gitlab `
--volume /e/volumes/gitlab/data:/var/opt/gitlab `
gitlab/gitlab-ce:18.5.4-ce.0

The same command works fine with Docker Desktop (E is an external disk drive available to Windows host)

What goes wrong

So far, we have the following flow:

  • GitLab requires real Linux filesystem permissions and ownership
  • During startup, it runs chown and chgrp on its data directories
  • Windows filesystems (NTFS) do not support Linux UID/GID ownership
  • WSL2 cannot translate these permission changes correctly
  • The operation fails, and GitLab refuses to start

If both Podman and Docker are based on WSL2, why does Docker run GitLab on an E: drive without breaking a sweat? The root cause is the difference in how Docker and Podman translate file permissions.

Docker: if GitLab calls chgrp, WSL’s drvfs layer intercepts the call. It doesn’t actually change the Windows folder, but it records the “permission change” in a hidden metadata area (NTFS Extended Attributes).

/etc/wsl.conf content of the docker desktop engine:

[automount]
root = /mnt/host
options = "metadata"
[interop]
enabled = true

When metadata is enabled as a mount option in WSL, extended attributes on Windows NT files can be added and interpreted to supply Linux file system permissions.

Podman: mounts Windows drives using the standard WSL2 9p protocol and drvfs driver (as Docker actually) without the complex metadata mapping enabled by default. When GitLab/your app tries to set its required ownership, the mount simply refuses, causing the container to crash

Here is an output for E disk drive mount from the podman machine:

mount | grep " /mnt/e "
E:\ on /mnt/e type 9p (rw,noatime,aname=drvfs;path=E:\;uid=1000;gid=1000;symlinkroot=/mnt/,cache=5,access=client,msize=65536,trans=fd,rfd=5,wfd=5)

there is no metadata option for the mount because of such simple wsl.conf:

[user]
default=user

Solution

The easiest solution here is to use named volumes (universal and faster) or a bind mount (if Docker is used; slower); custom wsl.conf and bind mount (if Podman is used; slower)

Named volumes:

podman run --detach --hostname gitlab.example.com `
--env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.example.com'" ` 
--publish 443:443 --publish 80:80 --publish 22:22 ` 
--name gitlab --restart always ` 
--volume gitlab-config:/etc/gitlab `
--volume gitlab-logs:/var/log/gitlab ` 
--volume gitlab-data:/var/opt/gitlab `
gitlab/gitlab-ce:18.5.4-ce.0

and the data will be stored at /var/lib/containers/storage/volumes (podman machine in this example):

Can be accessed from Windows Explorer as well:

  • Docker: \\wsl$\docker-desktop\mnt\docker-desktop-disk\data\docker\volumes
  • Podman: \\wsl$\podman-machine-default\var\lib\containers\storage\volumes

Bind mounts:

docker run --detach `
  --hostname gitlab.example.com `
  --publish 443:443 --publish 80:80 --publish 22:22 `
  --name gitlab-bind-mount `
  --restart always `
  --volume /e/volumes/gitlab/config:/etc/gitlab `
  --volume /e/volumes/gitlab/logs:/var/log/gitlab `
  --volume /e/volumes/gitlab/data:/var/opt/gitlab `
  gitlab/gitlab-ce:18.5.4-ce.0

Custom wsl.conf (podman):

[automount]
options = "metadata"

[user]
default=user

[interop] enabled=true is not actually required since it’s true by default, then restart podman and try podman run again


Docker and Podman use different WSL default configurations. Docker tolerates emulated ownership changes by enabling the metadata option out of the box.

Podman, on the other hand, does not rely on this metadata and expects real Linux filesystem behavior. It is also daemonless and lighter than Docker—but that’s a story for another blog post.