Category: DevOps

Argo CD Login and API: received unexpected content-type or TLS handshake timeout

My Argo CD is running on AWS EKS and is exposed via a standard Kubernetes Ingress (traefik class), meaning it also interacts with AWS ELB. Additionally, the server.insecure parameter in the Argo CD server is set to “true” (configured in the argocd-cmd-params-cm ConfigMap in Kubernetes), with TLS termination happening on the ingress side.

There are no issues with the Argo CD UI. However, I am unable to access the Argo CD API using simple curl requests or the Argo CD CLI. I keep receiving errors related to content-type and TLS handshake failures:

 argocd login argo.example.com --grpc-web --insecure --skip-test-tls
 FATA[0036] rpc error: code = Unknown desc = Post "https://argocd.example.com/session.SessionService/Create": net/http: TLS handshake timeout 
FATA[0045] rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"

All requests are being sent from a WSL instance (Ubuntu 22.04). Note that I have no issues accessing the API when using port forwarding or when connecting from the management partition (local machine).

I was about to give up, but then I decided to check the MTU size.

Get-NetIPInterface -AddressFamily IPv4 | Sort-Object -Property NlMtu | Select ifIndex, InterfaceAlias, NlMtu -first 5

ifIndex InterfaceAlias            NlMtu
------- --------------            -----
     19 Ethernet 3                 1392
     20 Local Area Connection* 1   1500
      8 Ethernet (WSL)             1500

Ethernet 3 is my VPN interface, and the API is only reachable through this interface.

Ethernet is the interface that WSL uses, so an MTU mismatch is occurring.

The solution is to adjust the MTU to match 1392 (the exact value may vary).

In your WSL instance, run the following:

ifconfig # to list interfaces. note your main interface (eth0 typically)
sudo ifconfig eth0 mtu 1392 # to change MTU size

I hope it helps!

Git: clone succeeded, but checkout failed

Have you ever faced any issues with git clone? Personally, I can’t recall any significant or memorable problems I’ve encountered while cloning remote repositories. Typically, the issues were related to authentication or network connectivity. Therefore, there was nothing particularly special to write about. However, as you work with different environments, the chances of coming across something interesting enough to share increase, even though it might be obvious to some.

Let’s take a simple example: you’re trying to clone an existing repository, which was created by someone else. The repository had already been filled out with files you need. Assuming you have credentials in place, you run git clone <repo’s url> on your Windows machine and get the following:

I hid the error message. I’ll reveal it later

What could go wrong? The cloning process succeeded, indicating that the issue is not related to Git credentials or network connectivity. However, the checkout process failed. What does this mean? It means that if you navigate to the folder of the cloned repository in the explorer, you won’t find any files written to the disk. Now, let me reveal the full error message, which is straightforward:

error: invalid path 'config/app1/application-staging.yml '
fatal: unable to checkout working tree

Found a “root cause”? There is the whitespace at the end of the filename.

However, you may wonder, since the repository was pre-created and used by other people, how did this happen? You’re correct to question that.

The reason is that Windows doesn’t support trailing space characters and automatically removes them when saving a file (you can read more about it here). On the other hand, Linux does support both leading and trailing whitespaces in filenames.

“file1.txt” and “file1.txt ” are two different files actually

Git knows about these limitations and has a special config setting to control it:

core.protectNTFS

If set to true, do not allow checkout of paths that would cause problems with the NTFS filesystem, e.g. conflict with 8.3 “short” names. Defaults to true on Windows, and false elsewhere.

The reason why other people can clone the repo without issues is that core.protectNTFS is set to false (manually or because of underlying OS)

So, to clone the repo on Windows you can use the following command:

get clone -c core.protectNTFS=false <repo url>
and now you can fix the wrong filename and sync with remote repo

As a summary, I would advise all developers and DevOps engineers to strictly avoid using trailing or leading spaces in filenames altogether. By doing so, we can eliminate the potential conflicts and issues that may arise from incompatible behaviors between different operating systems.