"kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

Mon Jun 15 16:10:47 GMT 2020

On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:

>
> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis <
> Brian.Inglis@systematicsw.ab.ca> wrote:
>
>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
>> > Brian.Inglis@systematicsw.ab.ca> wrote:
>> >
>> >> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>> >>>> On 14.06.2020 08:12, David Karr wrote:
>> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>> >>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
>> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>> >>>>>>> exec" to open a shell in a container or directly execute a shell
>> >>>>>>> command.
>> >>>>>>> This has worked perfectly fine for a long time.
>> >>>>>>> A couple of days ago, I discovered that all of these attempts were
>> >>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>> kubectl
>> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>> upgrade,
>> >>>>>>> but it's entirely possible.
>> >>>>>>> A colleague of mine has a very similar desktop configuration
>> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>> shows
>> >>>>>>> the resulting "curl" command that it runs. I tried that resulting
>> >>>>>>> command, and it results in the same response. I then tried
>> updating
>> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>> >> entire resulting "kubectl exec" command line and ran
>> >>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>> >>>>>>> I know I haven't provided much useful information yet. I wanted to
>> >>>>>>> get an initial response before I started providing those
>> diagnostics.
>> >>>>>>> Is there a clear issue here that I'm not aware of?
>> >>
>> >>>>> from where is kubectl coming from ?
>> >>>>> In cygwin I found only a kubectl.py in the ansible package
>> >>
>> >>>>>> It's from here:
>> >>>>>>
>> >>
>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>> >>
>> >>>> so it is NOT a cygwin program.
>> >>>> If the warning is coming about curl, it is likely
>> >>>> that using from cygwin you are using the cygwin curl
>> >>>> and from CMD the windows one
>> >>>> $ which -a curl
>> >>>> /usr/bin/curl
>> >>>> /cygdrive/c/WINDOWS/system32/curl
>> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>> >> smtps
>> >>>> telnet tftp
>> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>> >>>>
>> >>>> $ /usr/bin/curl -V
>> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>> zlib/1.2.11
>> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>> >>>> Release-Date: 2019-09-11
>> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
>> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>> >>>> TrackMemory UnixSockets
>> >>>> the support Forum https://discuss.kubernetes.io/
>> >>>> is probably the most indicate place for guidance
>> >>
>> >>> I thought it was obvious that it was not working because it was
>> calling
>> >> the
>> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
>> >>> And since I'm well aware of the k8s community, I already posted
>> questions
>> >>> about this in the appropriate place, before I posted here.
>> >>> What I was hoping to get here was some indication or thoughts on why a
>> >>> process using Windows curl doesn't have a problem, but does have a
>> >> problem
>> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>> is
>> >>> doing "wrong", it's just that it's doing something different.
>> >>> If it matters, the following is an elided version of the resulting
>> curl
>> >>> command:
>> >>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>> >> (windows/amd64)
>> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>> >>>
>> >>
>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>> >>> I can't tell from the logging what request body it sent. It's
>> possible it
>> >>> didn't send any.
>> >>
>> >> We can't tell, not knowing who you are, and from what you are posting,
>> >> what may
>> >> or may not be obvious to you, where you are seeing "Upgrade request
>> >> required",
>> >> or where that may be coming from.
>> >> That you need diagnostic help indicates that, what may appear obvious
>> to
>> >> you,
>> >> may not be the case, as it is often our assumptions which lead us
>> astray,
>> >> and we
>> >> get daily proof that we are imperfect, which is why most of seek to
>> talk
>> >> over
>> >> and explain our issues to inanimate objects or colleagues e.g. talk to
>> the
>> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>> and a
>> >> pelican awarded by former projects.)
>> >>
>> >> For help with Cygwin, we need to see *whole* commands and all the
>> output
>> >> between
>> >> the shell prompts, preferably with context, in this case including PATH
>> >> under
>> >> Cygwin and MS Windows and/or program paths typed or invoked.
>> >>
>> >> You may retain privacy and security by substituting variables e.g. env
>> >> vars $VAR
>> >> to sanitize sensitive information: to avoid problems I use a script to
>> do
>> >> so
>> >> that logs are never written containing sensitive content.
>> >>
>> >
>> > Acknowledged. The "Upgrade request required" is coming from the
>> kubernetes
>> > server.  At this point, I think I'm going to need to understand exactly
>> > what that message means (other threads talking about this didn't make
>> that
>> > clear). Someone in that community did respond to my question, not with
>> an
>> > answer, but with a query for more information. I'll see what I can find
>> out.
>>
>> A short Google search turned up that the message is apparently an HTTP
>> 400 Bad
>> Request response with Upgrade request message, meaning you need to use the
>> websocket API to talk to a kubernetes pod:
>>
>>
>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>> https://tools.ietf.org/html/rfc6455
>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>
>
> Yes, I saw that thread in my research on this, but note that that
> conclusion about using WebSocket is only from the first sentence of the
> answer, whereas by the end of the answer, including the comment, it's clear
> that this should be mitigated by just using "kubectl exec", which is what
> I'm doing. In any case, it appears that even kubectl uses curl under the
> covers. I would assume that kubectl attempts to mitigate the issues around
> not using WebSocket for this, but it seems like it's not working at least
> in my case.
>
> Another thought I had is whether I can coerce kubectl to use the Windows
> curl by setting the PATH just before calling kubectl.  This didn't help. Is
> there possibly something happening under the covers that would still make
> this execute the Cygwin curl?
>

I have managed to resolve this.  The challenge here is finding the correct
version of kubectl to use for the corresponding cluster version.  The rule
is that you have to use a version of kubectl within one minor version of
the cluster version.  We're using version 1.13.5 in the cluster.  I
originally saw this problem with version 1.17, then moved to 1.18, then to
1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
should have worked.  I don't know why it didn't.  I also have no idea why
it would work in a non-cygwin shell.