Tag: linux

Git: clone succeeded, but checkout failed

Have you ever faced any issues with git clone? Personally, I can’t recall any significant or memorable problems I’ve encountered while cloning remote repositories. Typically, the issues were related to authentication or network connectivity. Therefore, there was nothing particularly special to write about. However, as you work with different environments, the chances of coming across something interesting enough to share increase, even though it might be obvious to some.

Let’s take a simple example: you’re trying to clone an existing repository, which was created by someone else. The repository had already been filled out with files you need. Assuming you have credentials in place, you run git clone <repo’s url> on your Windows machine and get the following:

I hid the error message. I’ll reveal it later

What could go wrong? The cloning process succeeded, indicating that the issue is not related to Git credentials or network connectivity. However, the checkout process failed. What does this mean? It means that if you navigate to the folder of the cloned repository in the explorer, you won’t find any files written to the disk. Now, let me reveal the full error message, which is straightforward:

error: invalid path 'config/app1/application-staging.yml '
fatal: unable to checkout working tree

Found a “root cause”? There is the whitespace at the end of the filename.

However, you may wonder, since the repository was pre-created and used by other people, how did this happen? You’re correct to question that.

The reason is that Windows doesn’t support trailing space characters and automatically removes them when saving a file (you can read more about it here). On the other hand, Linux does support both leading and trailing whitespaces in filenames.

“file1.txt” and “file1.txt ” are two different files actually

Git knows about these limitations and has a special config setting to control it:

core.protectNTFS

If set to true, do not allow checkout of paths that would cause problems with the NTFS filesystem, e.g. conflict with 8.3 “short” names. Defaults to true on Windows, and false elsewhere.

The reason why other people can clone the repo without issues is that core.protectNTFS is set to false (manually or because of underlying OS)

So, to clone the repo on Windows you can use the following command:

get clone -c core.protectNTFS=false <repo url>
and now you can fix the wrong filename and sync with remote repo

As a summary, I would advise all developers and DevOps engineers to strictly avoid using trailing or leading spaces in filenames altogether. By doing so, we can eliminate the potential conflicts and issues that may arise from incompatible behaviors between different operating systems.

Simple PostgreSQL Backup Agent

Dockerized cron job to backup PostgreSQL database or multiple databases on different hosts. It’s based on Alpine docker image, so the image size is less than 11 Mb. The script can be also used without docker and docker compose or as a base for your own dockerized cron jobs. My general recommendation is to run docker container on your backup host to provide a kind of isolation from the management partition.

The script or “agent” does the following:

  • Reads content of /config/passfile to get pg_dump connection parameters
  • Verifies if the backup can be done by executing a dry run for each db
  • If the dry run is completed and plain format set, produces plain-text sql script and compresses it with gzip
  • If the dry run succeeds and custom format set, outputs a custom backup archive (more flexible and by default)
  • Cleans up the storage folder. Files older than 30 days are deleted
  • Redirects all cron job statuses to stdout
  • Keeps backup files under ./psql/backups/{hostname}/{dbname}/ on your host
  • Default settings: twice a day at 8:30 and 20:30 UTC; custom format; clean backups older than 30 days

Current limitations:

  • no encryption for specific databases (in to-do list)
  • no handling of wildcars in passfile (in to-do list)

Content

  • Dockerfile – describes docker image
  • docker-compose.yml – docker compose file to build and run agent service
  • /config/cronfile – cron job schedule settings
  • /config/passfile – PostgreSQL .pgpass actually
  • /config/psql_backup.sh – the script itself

Usage guide

  • check out the passfile and provide your own connection parameters
  • verify the cron job settings in the /config/cronfile
  • change make_backup function argument to set format output (plain/custom)
  • update cleaner function argument at the bottom of the script if necessary
  • edit dockerfile/docker-compose.yml or script itself if necessary
  • run docker compose build
  • run docker compose up -d
  • check out the stoud of the container to get the job’s status
  • TO RESTORE: use psql (if plain set) or pg_restore command (if custom format set)

Dockerfile

FROM alpine:3.16.2
LABEL AUTHOR="Roman Levchenko"
LABEL WEBSITE="rlevchenko.com"
RUN mkdir /etc/periodic/custom \
    && mkdir -p /backup/config \ 
    && touch /var/log/cron.log \
    && apk --no-cache add \
    postgresql14-client=14.5-r0 \
    bash=5.1.16-r2
COPY /config/cronfile /etc/crontabs/root
COPY /config/psql_backup.sh /etc/periodic/custom/backup
COPY ["/config/psql_backup.sh","/config/passfile","/backup/config/"]
RUN chmod 755 /etc/periodic/custom/backup \
    && chmod 0600 /backup/config/passfile
CMD ["-f","-l","8", "-L", "/dev/stdout"]
ENTRYPOINT ["crond"]

Script (excerpt)

# Clean old backup files
function cleaner()
{
set -o pipefail -e
	if [[ -n $(find $BACKUP_DIR \( -name "*.sql.gz" -o -name "*.custom" \) -type f -mtime +"$1") ]]; 
	then
		echo -e "\n${GREEN}[INFO]${OFF} ${BOLD}There are backup files older than $1 days. Cleaning up the following files:${OFF}"
		find $BACKUP_DIR \(-name "*.sql.gz" -o -name "*.custom" \) -print -type f -mtime +"$1" -exec rm {} \;
	else 
		echo -e "\n${GREEN}[INFO]${OFF} ${BOLD}There are no backup files older than $1 days. \nHave a nice day!${OFF}"
	fi
set +o pipefail +e
}

Result

Sample Output (w/error and success messages):

%d bloggers like this: