curl SSL CA Cert path error -- apt-get recommended pacakges

While setting up my dev environment to play around with golang I came across an issue when downloading packages using the go get package-name command

~# go get github.com/tools/godep
# cd .; git clone https://github.com/tools/godep /go/src/github.com/tools/godep
Cloning into '/go/src/github.com/tools/godep'...
fatal: unable to access 'https://github.com/tools/godep/': Problem with the SSL CA cert (path? access rights?)
package github.com/tools/godep: exit status 128

At first I thought it was something wrong with my boot2docker VM since I had disabled TLS after encountering issues when doing upgrades with the certificates getting corrupted. However, after chatting with some people at the #go-nuts IRC channel somebody suggested I could be under a MITM attack. I used Steve Gibson HTTPs fingerprint service and confirmed that nobody was attacking me.

To narrow down the possible causes of the issue I used curl to get github’s finger print from the following locations

  • mac
  • boot2docker
  • official ubuntu image
  • customized ubuntu image

Below are the results:

mac

Connected to github.com (192.30.252.130) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Server certificate: github.com
Server certificate: DigiCert SHA2 Extended Validation Server CA
Server certificate: DigiCert High Assurance EV Root CA

boot2docker

Connected to github.com (192.30.252.128) port 443 (#0)
successfully set certificate verify locations: CAfile: /usr/local/etc/ssl/certs/ca-certificates.crt
CApath: none
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server key exchange (12):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using ECDHE-RSA-AES128-SHA
Server certificate:
subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=5157550; street=548 4th Street; postalCode=94107; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
start date: 2014-04-08 00:00:00 GMT
expire date: 2016-04-12 12:00:00 GMT
subjectAltName: github.com matched
issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
SSL certificate verify ok.

Official Ubuntu Image

Hostname was NOT found in DNS cache
Trying 192.30.252.128...
Connected to github.com (192.30.252.128) port 443 (#0)
successfully set certificate verify locations:
CAfile: none  
CApath: /etc/ssl/certs
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS alert, Server hello (2):
SSL certificate problem: unable to get local issuer certificate
Closing connection 0

Customized Ubuntu Image

Connected to github.com (192.30.252.128) port 443 (#0)
successfully set certificate verify locations:
CAfile: none
CApath: /etc/ssl/certs
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS alert, Server hello (2):
SSL certificate problem: unable to get local issuer certificate
Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

Root cause of the problems

After checking the results it was clear that something I had done while customizing my Ubuntu image was the source of the CA store problems. The Dockerfile for my custom Ubuntu image was the following:

FROM ubuntu:14.04.3

RUN apt-get update

RUN apt-get install -y \
    curl git vim build-essential \
    —no-install-recommends 
    
.....

The culprit was the –no-install-recommends.

I first saw the –no-install recommends flag being used in the Dockerfile for the golang official image

FROM buildpack-deps:jessie-scm

# gcc for cgo
RUN apt-get update && apt-get install -y \
		gcc libc6-dev make \
		--no-install-recommends \
	&& rm -rf /var/lib/apt/lists/*

ENV GOLANG_VERSION 1.4.2

RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.src.tar.gz \
		| tar -v -C /usr/src -xz

RUN cd /usr/src/go/src && ./make.bash --no-clean 2>&1

ENV PATH /usr/src/go/bin:$PATH

RUN mkdir -p /go/src /go/bin && chmod -R 777 /go
ENV GOPATH /go
ENV PATH /go/bin:$PATH
WORKDIR /go

COPY go-wrapper /usr/local/bin/

The flag tells apt-get to only install the required packages and drop the recommended packages. When installing packages through apt-get there are three categories

  • required
  • recommended
  • suggested

By default the suggested packages are not installed and the recommended and required are. The flags are the following

  • –no-install-recommends – Do not consider recommended packages as a dependency for installing. Configuration Item: APT::Install-Recommends.
  • –install-suggests – Consider suggested packages as a dependency for installing. Configuration Item: APT::Install-Suggests.

Checking the package dependency for curl I found that the package that installs the CA root certificates was called ca-certificates and it was listed as recommended

curl package list

  • The following extra packages will be installed:
    • libasn1-8-heimdal libcurl3 libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libidn11 libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libroken18-heimdal librtmp0 libsasl2-2 libsasl2-modules-db libwind0-heimdal
  • Suggested packages:
    • krb5-doc krb5-user
  • Recommended packages:
    • ca-certificates krb5-locales libsasl2-modules

So to solve the issue I simply removed the –no-install-recommends flag from my custom Ubuntu image Dockerfile and I was able to download packages using go get