Showing posts with label Kubenetes. Show all posts
Showing posts with label Kubenetes. Show all posts

Thursday, March 8, 2018

Azure Container Instance (5): Building HTTPS Websites


HTTPS websites are more and more popular recently. Now you can also build websites with HTTPS connections via Azure Container Instance, with secret volumes and DNS Name Labels. This article is to demonstrate the steps about how to setup HTTS connections for a Node.js website.

Prerequisite: Have a Certificate

You may have a certificate for the SSL connection. If your DNS name label xyz, and your container group is going to be created in WestUS region, the fully qualified domain name looks like xyz.westus.azurecontainer.io and your certificate should match it. If you have a CName for your website, your certificate should match the CName.

Step 1: Build the Image

If you are going to build your website with Node.js, you may setup HTTPS connections with the following code:

const fs = require('fs');
const https = require('https');
const express = require('express');
const morgan = require('morgan');
const options = {
    pfx: fs.readFileSync('certificate.pfx'),
    passphrase: fs.readFileSync('certificatepassword.txt')
};

const app = express();
app.use(morgan('combined'));

app.get('/', (req, res) => {
    res.sendFile(__dirname + '/index.html')
});

var listener = https.createServer(options, app).listen(process.env.PORT || 443, function () {
    console.log('listening on port ' + listener.address().port);
});


Notice that the two files ‘certificate.pfx’ and ‘certificatepassword.txt’ are the certificate and its password. Before we start to run Node.js, these two files are copied from the path /mnt/secrets, as shown in Dockerfile:

CMD cp /mnt/secrets/sslcertificateData /usr/src/app/certificate.pfx && cp /mnt/secrets/sslcertificatePwd /usr/src/app/certificatepassword.txt && node /usr/src/app/index.js

We are going to pass these secrets from the Azure deployment template.

All other code for the Node.js website and the Docker file are shared at https://github.com/zhedahht/aci-ssl-helloworld. And the built image is shared at https://hub.docker.com/r/containerinstance/helloworld/.

Step2: Define the Deployment Template

The next step is to define Azure deployment template, and the following is a sample:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containergroupname": {
      "type": "string",
      "metadata": {
        "description": "Name for the container group"
      }
    },
    "containername": {
      "type": "string",
      "metadata": {
        "description": "Name for the container"
      },
      "defaultValue": "container1"
    },
    "imagename": {
      "type": "string",
      "metadata": {
        "description": "Name for the image"
      },
      "defaultValue": "containerinstance/helloworld:ssl"
    },
    "volumename": {
      "type": "string",
      "metadata": {
        "description": "Name for the secret volume"
      },
      "defaultValue": "volume1"
    },
    "dnsnamelabel": {
      "type": "string",
      "metadata": {
        "description": "The DSN name label"
      }
    },
    "sslcertificateData": {
      "type": "securestring",
      "metadata": {
        "description": "Base-64 encoded authentication PFX certificate."
      }
    },
    "sslcertificatePwd": {
      "type": "securestring",
      "metadata": {
        "description": "Base-64 encoded password of authentication PFX certificate."
      }
    },
    "port": {
      "type": "string",
      "metadata": {
        "description": "Port to open on the container and the public IP address."
      },
      "defaultValue": "443"
    },
    "cpuCores": {
      "type": "string",
      "metadata": {
        "description": "The number of CPU cores to allocate to the container."
      },
      "defaultValue": "1.0"
    },
    "memoryInGb": {
      "type": "string",
      "metadata": {
        "description": "The amount of memory to allocate to the container in gigabytes."
      },
      "defaultValue": "1.5"
    }
  },
  "variables": {},
  "resources": [
    {
      "name": "[parameters('containergroupname')]",
      "type": "Microsoft.ContainerInstance/containerGroups",
      "apiVersion": "2018-02-01-preview",
      "location": "[resourceGroup().location]",
      "dependsOn": [],
      "properties": {
        "containers": [
          {
            "name": "[parameters('containername')]",
            "properties": {
              "command": [],
              "image": "[parameters('imagename')]",
              "ports": [
                {
                  "port": "[parameters('port')]"
                }
              ],
              "resources": {
                "requests": {
                  "cpu": "[parameters('cpuCores')]",
                  "memoryInGb": "[parameters('memoryInGb')]"
                }
              },
              "volumeMounts": [
                {
                  "name": "[parameters('volumename')]",
                  "mountPath": "/mnt/secrets",
                  "readOnly": false
                }
              ]
            }
          }
        ],
        "osType": "Linux",
        "ipAddress": {
          "type": "Public",
          "dnsNameLabel": "[parameters('dnsnamelabel')]",
          "ports": [
            {
              "protocol": "tcp",
              "port": "[parameters('port')]"
            }
          ]
        },
        "volumes": [
          {
            "name": "[parameters('volumename')]",
            "secret": {
              "sslcertificateData": "[parameters('sslcertificateData')]",
              "sslcertificatePwd": "[base64(parameters('sslcertificatePwd'))]"
            }
          }
        ]
      }
    }
  ],
  "outputs": {
    "containerIPAddressFqdn": {
      "type": "string",
      "value": "[reference(resourceId('Microsoft.ContainerInstance/containerGroups/', parameters('containergroupname'))).ipAddress.fqdn]"
    }
  }
}

In the template above, the certificate is mounted as the file /mnt/secrets/sslcertificateData, and the password is mounted as the file /mnt/secrets/sslcertificatePwd. These files will be copied and pasted as /user/src/app/certificate.pfx and /user/src/app/certificatepassword.txt respondingly, and they are accessible to Node.js.

The DNS name labels is defined in the property dnsNameLabel of ipAddress. The fully qualified domain name is a concatenation of the DSN name label, the location and “azurecontainer.io”. 

The deployment template and the corresponding parameters are shared at https://github.com/Azure/azure-quickstart-templates/tree/master/201-aci-linuxcontainer-volume-secret.

Monday, August 7, 2017

Kubernetes Notes (3): Failed to Mount Azure Files


We may meet failures when we try to mount Azure files onto containers orchestrated by Kubernetes. Here is about how to resolve the failures.

The first error complains "No such file or directory" when describing the pod with kubectl, as shown in the following image:

The error complaining "No such file or directory"

The root cause of this issue is that the Azure file share declared in volumes.azureFile.shareName property of the pod deployment configure doesn't exit. Please go the Azure portal to create an Azure file share under your Azure storage account.

Another error complains "Permission denied" when describing the pod with kubectl, as shown in the screenshot below:

The error message complaining "Permission denied"

Of course, we will meet this error when the storage account name or key in the secret is incorrect. Kubernetes requires that secrets should be encoded with base64 algorithm. If we just copy the storage account names and keys into Kubernetes secrets, we will see such an error.

Many ones encode secrets with the Linux echo command and then pipe it into base64. Please make sure use the "-n" option with the echo command, otherwise we will meet the "permission denied" error. The following screenshot demonstrate that different encoded strings are returned when encoding with or without the "-n" option:

The "-n" option of echo command

Another somewhat counterintuitive issue which also triggers "Permission denied" error is that the location of storage account is different from the location of container hosts. Please go to the portal to check the location of the storage account and VMs.

Sunday, July 16, 2017

Kubernetes Notes (2): Node Prioritization on Resources


When there are multiple nodes with enough resources available to deploy pods, the Kubernetes scheduler selects the node with highest score. Let's discuss how Kubernetes prioritize nodes based on resources.

The Kubernetes scheduler has three algorithm related to resources. The first one is the least_requested algorithm, with which the Kubernetes scheduler tends to spread pods out and keep resource utilization rate on every node low. The algorithm looks like below:

// The unused capacity is calculated on a scale of 0-10
// 0 being the lowest priority and 10 being the highest.
// The more unused resources the higher the score is.
func calculateUnusedScore(requested int64, capacity int64, node string) int64 {
    if capacity == 0 {
        return 0
    }
    if requested > capacity {
        glog.V(10).Infof("Combined requested resources %d from existing pods exceeds capacity %d on node %s",
            requested, capacity, node)
        return 0
    }
    return ((capacity - requested) * 10) / capacity
}

allocatableResources := nodeInfo.AllocatableResource()
totalResources := *podRequests
totalResources.MilliCPU += nodeInfo.NonZeroRequest().MilliCPU
totalResources.Memory += nodeInfo.NonZeroRequest().Memory
cpuScore := calculateUnusedScore(totalResources.MilliCPU, allocatableResources.MilliCPU, node.Name)
memoryScore := calculateUnusedScore(totalResources.Memory, allocatableResources.Memory, node.Name)

The final score is the average of cpuScore and memoryScore. The code above shows that the nodes with lower resource utlization rate have higher score, then highter priority to deploy pods. If there are two nodes (with 2 CPU and 4 CPU respectively) available when scheduling a pod requesting 1 CPU, the least_requested algorithm tends to select the node with 4 CPU.

The most_requested algorithm behaves in the opposite way, with which the Kubernetes scheduler tends to deploy pods onto nodes with the highest resource utilization rate. The code to score nodes is below:

// The used capacity is calculated on a scale of 0-10
// 0 being the lowest priority and 10 being the highest.
// The more resources are used the higher the score is. This function
// is almost a reversed version of least_requested_priority.calculatUnusedScore
// (10 - calculateUnusedScore). The main difference is in rounding. It was added to
// keep the final formula clean and not to modify the widely used (by users
// in their default scheduling policies) calculateUSedScore.
func calculateUsedScore(requested int64, capacity int64, node string) int64 {
    if capacity == 0 {
        return 0
    }
    if requested > capacity {
        glog.V(10).Infof("Combined requested resources %d from existing pods exceeds capacity %d on node %s",
            requested, capacity, node)
        return 0
    }
    return (requested * 10) / capacity
}

If there are two nodes (with 2 CPU and 4 CPU respectively) available when scheduling a pod requesting 1 CPU, the most_requested algorithm tends to select the node with 2 CPU.

The third algorithm is balanced_resource_allocation, with which the Kubernetes scheduler tries to balance the utilization rates of CPU and memory. Its related code looks like:

allocatableResources := nodeInfo.AllocatableResource()
totalResources := *podRequests
totalResources.MilliCPU += nodeInfo.NonZeroRequest().MilliCPU
totalResources.Memory += nodeInfo.NonZeroRequest().Memory

cpuFraction := fractionOfCapacity(totalResources.MilliCPU, allocatableResources.MilliCPU)
memoryFraction := fractionOfCapacity(totalResources.Memory, allocatableResources.Memory)
score := int(0)
if cpuFraction >= 1 || memoryFraction >= 1 {
    // if requested >= capacity, the corresponding host should never be preferred.
    score = 0
} else {
    // Upper and lower boundary of difference between cpuFraction and memoryFraction are -1 and 1
    // respectively. Multilying the absolute value of the difference by 10 scales the value to
    // 0-10 with 0 representing well balanced allocation and 10 poorly balanced. Subtracting it from
    // 10 leads to the score which also scales from 0 to 10 while 10 representing well balanced.
    diff := math.Abs(cpuFraction - memoryFraction)
    score = int(10 - diff*10)
}

In the code above, it calculates the CPU and memory utilization rate first, and then their difference. The node with the highest resource utilization rate difference has the lowest priority.

The Kubernetes scheduler doesn't prefer nodes with 100% CPU or memory utilization rate. When a node with 100% CPU or memory utilization, its score is 0 and it is in the lowest priority to deploy pods.

When pods don't request resources explicitly (in Resources.Requests of deployment config), the Kubernetes scheduler treat them with 0.1 CPU and 200M memory requests by default when scoring nodes (non-zero.go):

// For each of these resources, a pod that doesn't request the resource explicitly
// will be treated as having requested the amount indicated below, for the purpose
// of computing priority only. This ensures that when scheduling zero-request pods, such
// pods will not all be scheduled to the machine with the smallest in-use request,
// and that when scheduling regular pods, such pods will not see zero-request pods as
// consuming no resources whatsoever. We chose these values to be similar to the
// resources that we give to cluster addon pods (#10653). But they are pretty arbitrary.
// As described in #11713, we use request instead of limit to deal with resource requirements.
const DefaultMilliCpuRequest int64 = 100             // 0.1 core
const DefaultMemoryRequest int64 = 200 * 1024 * 1024 // 200 MB

The Kubernetes scheduler has --algorithm-provider to config the algorithms to prioritize nodes, which has two options DefaultProvider and ClusterAutoScalerProvider. Both options include the balanced_resource_allocation algorithm. The only difference between these two options is that DefaultProvider uses the least_requested algorithm, while ClusterAutoScalerProvider utilizes the most_requested algorithm.

AKS (1) - Five seconds latency when resolving DNS

We intermittently meet 5s latencies in an AKS clusters with CNI when it’s resolving DNS. This article is to summarize what we have learned...