Cloud-native development grows up

When you book an airline ticket, apply for a new passport, access your insurance documents or bank account, you’re typically relying on software built by enterprise developers in corporate labs and cities around the world.

Enterprise developers are busy people. Customer’s expectations are higher than they’ve ever been for fast, mobile, and secure access to information. We already accept that microservices and cloud-based solutions offer the only real flexible, scalable future for the enterprise. Yet only 20% of enterprise software has moved to the cloud.

What is preventing more companies from moving to the cloud? Enterprise developers are often pressed for time to learn on the job. Even if they have time, it can be difficult knowing where to start with cloud-native development. The technology is constantly evolving, and opinions on best tools and approaches vary even within small organizations.

In this blog post, we introduce you to new cloud-native products and open source projects from IBM that simplify your journey to the cloud.

Cloud Pak for Applications

Cloud Pak for Applications aims to clear some of the mystery around Cloud Native development by:

  • Bundling the best in class libraries and frameworks for developing secure, fast and scalable solutions
  • Instilling customizable and consistent access to frameworks approved by an organization.

The image below shows the underlying technology that is included in Cloud Pak for Applications:

Cloud Pak Architecture

Cloud Paks for Applications contains a few big components, one of which is Kabanero – a collection cloud-native tools and libraries that we think are essential for cloud-native development.

We’re introducing a new collection of code patterns, articles and tutorials that gently introduce the concepts of Kabanero within Cloud Pak for Apps as a smart, disciplined and consistent approach to creating cloud native applications in the enterprise.

Because Kabanero is a core component of our cloud-development offering, let’s take a closer look at the underlying technology.

Kabanero development technologies

Kabanero is the open source foundational layer of Cloud Paks for Applications. Kabanero itself is made up of accepted, best-in-class cloud technologies which are all open source. You can see a graphical representation of many of the important technologies below:

Kabanero Architecture

One of the special ingredients in Kabanero is Appsody which uses technology stacks and templates to create a disciplined and consistent approach to developing apps within an enterprise organization.

Our approach to creating developer resources around Kabanero and Cloud Paks for Apps is to focus on workflows using the cloud DevOps components, providing tutorials around them, and code patterns that can be cloned and explored as reference models. In our first collection of developer resources, we’re sharing an Appsody code pattern that walks through the basics of creating an application that has two microservices – with presentation and business logic, as well as digging into approaches for using Appsody in your own projects.

Building with Appsody for consistent results

Appsody is an open source project that simplifies and controls cloud-native application development. Appsody’s primary component is a stack, which builds a pre-configured Docker image that developers can immediately use to create applications in a cloud environment. Appsody allows stack builders to decide which parts of the users’ resulting application images are fixed (a set of technology choices and configurations defined by the stack image) and which parts stack users can modify/extend (templates).

One way to think about Appsody is that it can give developers the advantages of a Platform as a Service (PaaS) environment (in terms of not having to worry about installing and configuring the underlying technology components), while allowing architects the flexibility to define those technology components using Docker images.

Appsody stacks

An Appsody stack represents a pre-configured set of technologies aimed at simplifying the building of a particular type of cloud native application. This might include a particular environment (for example, node.js, or perhaps python-flask), combined with integrated choices for monitoring, logging etc. Stacks are published in stack repositories, which can either be public or private to an enterprise. Developers can then use the Appsody CLI to pull in the appropriate stack for the application they are building. Kabanero contains all the tools for using and contributing to public stack repositories, as well as a set of curated stacks suitable for the enterprise.

Appsody goes even further than simplifying the use of pre-configured technologies. It enables developers to create and test applications within a local containerized environment from the start using Rapid Local Development Mode. After those initial tests are run, developers can then deploy the final application to cloud-based testing and production clusters. Developing in containers from the start reduces the likelihood of subtle problems being introduced when containerization is added late in the development process.

Appsody templates

Appsody stacks come with one or more templates. A template represents a starter application using that stack and comes ready to be run and deployed. Developer can modify the template to build out their application.

The following image shows the flow of how a developer uses Appsody to pull down and modify a stack, build it and then deploy it to a remote Kubernetes cluster.

Appsody Architecture

The above flow shows the manual deployment to a Kubernetes cluster. In more production-orientated environments, GitOps might trigger the build and deploy steps and Tekton Pipelines would drive the deployment. Kabanero Collections, which is part of Cloud Pak for Applications, brings together Appsody stacks, GitOps, and Tekton Pipelines to provide an enterprise-ready solution for grown-up cloud-native application development and deployment.

Ready to start?

Now that you understand the technology that underlies IBM Cloud Pak for Applications, you’re ready to start exploring the content that we’ve created. We’ve selected two different paths to help you get started with Cloud Pak for Applications:

Anton McConville
Henry Nash
Brad Topol

IBM Cloud Pak for Applications eases digital transformation for developers

The IBM Cloud Pak for Applications is the first of five Cloud Paks that IBM released in response to our clients’ need for tighter integration across IBM’s portfolio to enable your journey to cloud. Cloud Pak for Applications is a secure, complete, containerized set of capabilities for developers, solution architects, and operators that addresses the entire application lifecycle: architecture, development, management, and DevOps.

Most organizations still need to move 80% of their infrastructure to the cloud. To accelerate this modernization journey, Cloud Pak for Apps connects the agility and DevOps of cloud-based modern application development to the functionality and dependability of existing Java-based workloads. By leveraging Red Hat OpenShift, the IBM Cloud Pak for Applications provides a hybrid, multicloud foundation built on open standards, freeing workloads and data to run anywhere.

To help developers get started, we’re excited to announce the Cloud Pak for Applications Developer Hub, written by developers, for developers. This hub offers everything you need to migrate existing Java-based applications, as well as to build new and modify existing cloud-native applications.

Access step-by-step tutorials, code patterns, community, and ongoing education and support from IBM’s subject matter experts in application migration and modernization.

We’ve also created learning paths where we curated lists of content based on whether you just want introductory content, are a developer, or are a solution architect. Each path begins with introductory material, and continues through intermediate, advanced, and expert-level sets of content.

The best part is, you can — and should — start now! The technology is rapidly evolving, so we’ll continuously update the learning paths to keep you up to date and ease your transformation journey.

Willie Tejada

Applications are moving to the cloud. It’s time for developer tools to move, too

The cloud developer landscape is changing rapidly. Every day, there are new tools, new patterns, new technologies, and new frameworks for developers to learn and use. In cloud-native development, cloud architectural patterns like microservices require that developers rethink how they develop applications. Testing environments are more complex. Requirements for consistency in production environments and even basic setup and configuration for developer environments can be time-consuming operations. Developers need better tools to keep up with this quickly changing landscape.

That’s why we’ve joined a new working group at the Eclipse Foundation — the Eclipse Cloud Development Tools Working Group — whose goal is to accelerate the creation of those cloud-based developer tools. This is a vendor-neutral working group with members from a broad set of companies who work together to define standards for creating and using cloud-based developer tools.

We are working together to:

  • Define de-facto standards to drive broad adoption of cloud IDEs and container-based development tools
  • Enable an ecosystem for extenders and developer tool providers via these standards
  • Integrate with key enablers for cloud native development, CI, and test automation

Why do standards matter?

While standards may sound counter to rapid innovation, they are key enablers of extensibility, and interoperability. There are de-facto standards emerging for cloud-based tools in workspace definitions, extensions for languages support, tracing, and debugging. Our work group focuses on getting developers to adopt these standards. In turn, this will make the cloud-based developer tools interoperable with other cloud technologies. I believe that once we establish cloud development tools standards, it will enable a marketplace ecosystem for extensions which in turn benefits users and our customers.

Cloud native is a new way for developers to think

Developers are always trying to develop applications faster. Cloud-native tools, running in the cloud, will give developers new capabilities that leverage and exploit cloud capabilities from the very start of their development process. In turn, this lets developers test, build, monitor, and deploy applications faster in an environment that mirrors their production systems. This high fidelity development environment will enable productivity, so developers can focus on their work and innovate faster.

Some use cases where I can see how cloud-native developer tools will speed and improve development include:

  • Simpler setup and installation of development dependencies
  • Accessible, easy-to-use tools for A/B testing, always-on monitoring, and testing experimental aspects of development
  • Browser-based development to lower the barriers of entry for developers working in the cloud

The way that this will enhance how developers can get started and quickly create, test, monitor, and deploy applications is hard to overstate.

An example of cloud-native tools that we’ll champion in this group

One of the Eclipse projects that I’m excited to see championed through this new workgroup is Eclipse Codewind. This tool is an IDE extension that bundles performance and monitoring tools and enables you to develop in containers within your own IDE. You can make changes against all of your apps using the simple extension and instantly see how those changes perform in your development cluster. Tools like Codewind will help you develop better-performing, error-free applications faster than ever.

The working group is just getting started, and their are a lot of great things we are going to accomplish. The participants are from leading companies and their developers work in many exciting projects at Eclipse, so working together on standards will benefit all of our companies.

Get involved

If you are interested in promoting interoperative tools that run in the cloud, standards that allow those tools to be extended into any cloud, and an ecosystem to support the adoption of the standards and cloud-native hosted tools, view our Charter and ECD Working Group Participation Agreement (WPGA), or join the ECD Tools mailing list.

If you’re a developer who wants to enhance cloud-native development tools, check out the projects at the Eclipse Foundation. I’d say that for cloud tools as well as other projects, there a a bunch of great projects doing innovative things in open source at Eclipse. It’s a great way to work, and a great group of developers driving key innovations.

John Duimovich

Flying Kubernetes webinar: Key concepts explained with drones

Kubernetes is one of the fastest-growing technologies in the industry, and it’s not hard to tell why. It provides an isolated and secure app platform for managing containers, transforming both application development and operations for your organization.

But are you worried that Kubernetes is complex and difficult to learn?

To make learning Kubernetes concepts a little more fun, our team at IBM built a “Kubefly” demo to teach and explain core Kubernetes concepts by using a swarm of flying drones. The drones showcase concepts like pods, replica sets, deployments, and stateful sets by reacting to configuration changes on our Kubernetes cluster. For example, after a Kubernetes application is deployed, a few drones take off. Each one represents a pod in the deployment. If one of the Kubernetes pods is killed, the drone lands, and another takes its place, because Kubernetes uses a declarative model to always attempt to match the specified state.

Kubefly Kubernetes drones

To see these drones in action, register for the upcoming webinar with Jason McGee, Chief Technology Officer for IBM Cloud, and Briana Frank, Director of Product Management for IBM Cloud. The webinar demonstrates Kubernetes concepts and introduces basic benefits of the Istio service mesh.



If you want to learn a bit more about the drones behind the project, see the Flying Kubernetes with the Kubefly project blog post and video from earlier this year.

Belinda Vennam

Istio 1.3 is out: Here’s what it means for you

Companies with large, monolithic applications are increasingly breaking these unwieldy apps into smaller, containerized microservices. Microservices are popular because they offer agility, speed, and flexibility, but they can be complex, which can be a hurdle for adoption. And having multiple microservices, rather than a single monolith, can increase the attack surface of the app.

Istio gives control back to app developers and mesh operators. Specifically, Istio is an open source service mesh platform that ensures that microservices are connecting to each other in a prescribed way while handling failures. With Istio, it’s easier to observe what is happening across an entire network of microservices, secure communication between services, and ensure that policies are enforced.

A new release of Istio, version 1.3, makes using the service mesh platform even easier.

Istio 1.3 improves usability

Because Istio has so many features, it was more complex than other open service mesh projects I’ve tried. If we were going to accomplish our goal of making Istio the preferred service mesh implementation, we had to make it easier for other developers to use.

Specifically, we had to simplify the process for developers to move microservices to the Istio service mesh, regardless of whether they wanted to leverage security, traffic management, or telemetry first. We created a User Experience Work Group that engaged with the community to improve Istio’s user experience. Through community collaboration across many work groups and the Envoy community, I’m excited to see these changes in Istio 1.3:

  • All inbound traffic will be captured by default. There is no need to declare containerPort in your Kubernetes deployment YAML for Istio to indicate the inbound ports you want your Envoy sidecar to capture.
  • A single add-to-mesh command in the CLI adds existing services to Istio mesh regardless of whether the service runs in Kubernetes or a virtual machine.
  • A describe command that allows developers to describe the pod and service needed to meet Istio’s requirements and any Istio-associated configuration.
  • Automatic protocol detection is implemented and enabled by default for outbound traffic, but disabled for inbound traffic to allow us to stabilize this feature. You will still need to modify your Kubernetes service YAML to name or prefix the name of the service port with the protocol for v1.3, but I expect this requirement to be eliminated in a future release.

Refer to Istio 1.3’s release blog and release note for more details about the release.

Istio 1.3 in action

A little over a year ago, I tried to move the popular Kubernetes guestbook example to run in the Istio mesh. It took a few days because I didn’t follow the documentation closely and discovered the proper documentation only after I finished. Injecting a sidecar didn’t cause me a problem; I was tripped up by the list of requirements to pods and services.

Using this same example, let’s see how using Istio 1.3 simplified this process.

I had already deployed the upstream guestbook sample in my Kubernetes cluster in IBM Cloud (1.14.6) by using the guestbook-all-in-one.yaml file. I uncommented out line 108 to use type: LoadBalancer for the front-end service:

$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
frontend-69859f6796-4nj7p      1/1     Running   0          49s
frontend-69859f6796-772sw      1/1     Running   0          49s
frontend-69859f6796-n67w7      1/1     Running   0          49s
redis-master-596696dd4-ckcj4   1/1     Running   0          49s
redis-slave-96685cfdb-8cfm2    1/1     Running   0          49s
redis-slave-96685cfdb-hwpxq    1/1     Running   0          49s

I used the quick start guide to install Istio 1.3. The community is making progress to reduce the number of custom resource definitions (CRDs), and we are down to 23. We will continue to reduce the number of CRDs based on the Istio features the users install.

Once installed, I used the add-to-mesh command to add the frontend service to the Istio mesh:

$ istioctl x add-to-mesh service frontend
deployment frontend.default updated successfully with Istio sidecar injected.
Next Step: Add related labels to the deployment to align with Istio's requirement: https://istio.io/docs/setup/kubernetes/additional-setup/requirements/

As the result, my frontend pods had sidecar injected and running.

$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
frontend-69577cb555-9th62      2/2     Running   0          25m
frontend-69577cb555-pctrg      2/2     Running   0          25m
frontend-69577cb555-rvjk4      2/2     Running   0          25m
redis-master-596696dd4-dzf29   1/1     Running   0          26m
redis-slave-96685cfdb-2h789    1/1     Running   0          26m
redis-slave-96685cfdb-7sp6p    1/1     Running   0          26m

I described one of the frontend pods to see if the pod and associated frontend service met the requirements for Istio.

$ istioctl x describe pod frontend-69577cb555-9th62
Pod: frontend-69577cb555-9th62
   Pod Ports: 80 (php-redis), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: frontend
   Port:  80/UnsupportedProtocol
   80 is named "" which does not follow Istio conventions
Pilot reports that pod is PERMISSIVE (enforces HTTP/mTLS) and clients speak HTTP

The code clearly told me what was missing! I needed to name the port for its protocol, which is HTTP in this case. The version label for telemetry is optional for me as the frontend service only has one version.

I edited the frontend service using kubectl.

$ kubectl edit service frontend

I added a line to name the service, using name: http and saved the change.

…
  ports:
  - nodePort: 30167
    port: 80
    protocol: TCP
    targetPort: 80
    name: http
…
service/frontend edited

I repeated the same istioctl x add-to-mesh service redis-master and istioctl x add-to-mesh redis-slave commands to add redis-master and redis-slave to the mesh. For redis, I just used the default tcp protocol, so there was no need to name the port.

$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
frontend-b8595c9f-gp7vx        2/2     Running   0          51m
frontend-b8595c9f-pr5nb        2/2     Running   0          51m
frontend-b8595c9f-q7fx9        2/2     Running   0          50m
redis-master-5589dc575-bqmtb   2/2     Running   0          51m
redis-slave-546f8d974c-gq4sn   2/2     Running   0          50m
redis-slave-546f8d974c-qjjwl   2/2     Running   0          50m

I added the frontend, redis-master and redis-slave services to the mesh! I visited the guestbook app using the load balancer IP:

$ export GUESTBOOK_IP=$(kubectl get service frontend -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ curl $GUESTBOOK_IP

I used the istioctl dashboard grafana command to launch Grafana, which served up the guestbook service metrics. Grafana metrics

From there, I used istioctl dashboard jaeger to launch Jaeger, which gave me the guestbook traces for each guestbook request. Note that they are individual trace spans–not correlated. I expected the individual trace spans because I would need to propagate the trace headers so that multiple trace spans could be tied to each individual request.

Grafana metrics

After that, I used istioctl dashboard kiali to launch Kiali, which allowed me to visualize the guestbook app.

Grafana metrics

I was able to observe the microservices at the app layer with Grafana, Jaeger, or Kiali, without needing to modify the service code or rebuild the guestbook container images! Depending on your business needs, you can either secure your microservices or build resilience into your microservices or control traffic by shifting as you develop newer versions of your microservices.

I believe the community is committed to make Istio easier to use as developers move their microservices into the mesh or out of the service. Live editing the service will not be necessary once the intelligent protocol detection feature is enabled by default for inbound traffic. This feature should be added soon.

Istio’s open source strength

Istio’s open architecture and ecosystem combine to make the technology effective. There is no vendor lock-in limiting the types of services you can use with it. Plus, Istio can run in any cloud model — public, private, on-premises, or a hybrid cloud model.

Istio was founded by IBM, Google, and Lyft in 2017, and a thriving community has grown around it to include other partners, such as Cisco, Red Hat, Pivotal, Tigera, Tetrate, and more. With over 400 contributors from over 300 companies at the time of this writing, the Istio project benefits from an active, diverse community and skill set.

Istio and IBM

IBM has been involved with Istio from before it was released to the public, with IBM donating our Amalgam8 project into Istio. As the second top contributor to the open project, IBMers now sit on the Istio Steering Committee and the Technical Oversight Committee, and co-lead the Environment workgroup, Build/Test Release workgroup, Performance and Scalability workgroup, User Experience workgroup, and Docs workgroup.

Why all the interest in Istio? With development teams looking to quickly scale, they need tools that can free them up to innovate and simplify how to build and deploy apps across environments — and that’s why IBM continues to invest in Istio.

Multiple cloud providers offer a managed Istio experience to simplify the install and maintenance of the Istio control plane. For example, IBM Cloud has a managed Istio offering that enables you to install Istio with a single action.

We know that our clients need to modernize their apps and app infrastructures, and we believe that Istio is a critical technology to help them do this safely, securely, and — with the new changes in the 1.3 release — easily.

Get involved with Istio

Because Istio is open source, we rely on an active community of developers to help improve, secure, and scale the tech. Here are a few ways for you to get involved:

Lin Sun is a Senior Technical Staff Member and Master Inventor at IBM. She is a maintainer on the Istio project and also serves on the Istio Steering Committee and Technical Oversight Committee.

Lin Sun

Microsurvival Part 4: On to Kubernetes

Note: This blog post is part of a series.

Hey Appy! I’m glad you’re back. Before we get started, I wanted to share with you a poem I’ve been working on.

Ops teams were on their knees.

Ops teams were in tears.

But Kubernetes came in sight.

Kubernetes became their guide.

So, what’d you think? Why are you so interested in the ceiling all of a sudden? Hmm, I’ll take that as a sign to continue working on my poetry writing skills.

Anyway, our last conversation covered container technologies. Now, I think it’s time to dive into Kubernetes, don’t you think?

What is Kubernetes?

Where do I begin? Let’s start with the meaning of Kubernetes. Kubernetes is a Greek word that means helmsman. Now, why do we need it? As applications like you grow and the number of components increase, the difficulty of configuring, managing, and running the whole system smoothly also increases. And since humans always tried to automate difficult and repetitive tasks, Kubernetes was created.

Kubernetes is one of the many container orchestrators out there that runs and manages containers. What does a container orchestrator do? It helps the operations team to automatically monitor, scale, and reschedule containerized applications inside a cluster in the event of hardware failure. It enables containerized applications to run on any number of computer nodes as if all those nodes were a single, huge computer. That makes it a whole lot easier for both developers and operations team to develop, deploy, and manage their applications. Your parents, Dev and Ops, would surely agree with this magic.

Kubernetes architecture

Next is the Kubernetes architecture. A Kubernetes cluster is a bunch of master nodes and worker nodes. A master node manages worker nodes. You can have one master node or more than one if you want to provide high availability. The master nodes provide many cluster-wide system management services for the worker nodes, and the worker nodes handle our workload. However, we won’t be interacting with them a lot (not directly at least). To set your desired state of the cluster, you need to create objects using the Kubernetes API. We can use kubectl to do that, which is the command-line interface of Kubernetes.

Here, let me draw you a picture:

image

The master node consists of basically four things:

  1. etcd is a data store. The declarative model is stored in etcd as objects. For example, if we say we want five instances of a certain container, that request is stored in the data store.
  2. Kubernetes controller manager watches the changes requested through the API server and attempts to move the current state of the cluster towards the desired state.
  3. Kubernetes API server validates and configures data for the API objects, which include pods, services, replication, controllers, and more.
  4. Kubernetes scheduler takes charge of scheduling pods on nodes. It needs to consider a lot information, including resource requirements, hardware/software constraints, and many other things.

Each worker node has two main processes running on them:

  • kubelet is something like a node manager. The master node talks to the worker nodes through kubelet. The master node tells kubelet what to do, and then kubelet tells the Pods what to do.
  • kube-proxy is a network proxy that reflects Kubernetes networking services on each node. When a request comes from outside of the cluster, the kube-proxy routes that request to the specific pod needed, and the pod runs the request on the container.

Now this picture has more details, but you can see how everything works together:

image

We use Kubernetes API objects to describe how our cluster should be, what applications to run in it, which container images to use, and how many of them should be running.

Get to know the following basic Kubernetes objects:

  • Pods, are the smallest deployable units of computing that can be created and managed in Kubernetes. Containers of an application run inside these pods.
  • Pods are mortal and are created and destroyed dynamically when scaling. For the pods to communicate with each other, we need services. A service is an abstraction which defines a logical set of pods and how to access them.
  • Volume is an abstraction that solves two problems. The first problem is that all the files inside a container are lost when it crashes. The kubelet restarts the container, but it is a new container with a clean state. The second problem is that two containers running in the same pod often share files.
  • Namespace lets you create multiple virtual clusters (called namespaces) backed by the same physical cluster. You use them in huge clusters with many users belonging to multiple teams or multiple projects.

Now, pay attention to the controllers, which build upon the basic objects to give us more control over the cluster and provide additional capabilities:

  • ReplicaSet ensures a set number of replica pods are running at any given time.
  • After you define a desired state in a deployment object, the deployment controller changes the current state to the desired state at a controlled rate.
  • A DaemonSet ensures that all (or some) nodes run the specified pod. When more nodes are added, pods are added to them.
  • Jobs create one or more pods and after they are successfully completed, the job is marked as complete.

Okay, I know that was a lot of information. But, guess what? This is far from enough to get a full idea of Kubernetes! However, it is enough to get you off to a good start. Don’t you agree, Appy?

Hey! Are you dozing off? Wow, maybe lectures aren’t your thing. Let’s try a hands-on approach.

Lucky for you, there are several labs that can help you understand the core concepts of Kubernetes that I just described. These labs only require you to have an IBM Cloud account. You can then create a free Kubernetes cluster to play with.

I hope you tell your parents, Dev and Ops, what you have learned so far from our meeting today and from the labs after you complete them. I am sure both of them will be happy to hear about it. Don’t forget to deliver my regards as well.

Have a great adventure, Appy! I know you’re off to a good start.



A previous version of this post was published on Medium.

Amro Moustafa

Microsurvival Part 3: Hello Docker

Note: This blog post is part of a series.

Hello, Developer! I am so glad you could join me today. I’m extremely happy that your child Appy told you about what we discussed, but even happier that you suggested meeting today. I know you want the best for Appy, to look good and get along with others. We’ve been talking about an appropriate and safe environment for Appy, especially as Appy continues to grow and mature.

So, you’re trying to work with Docker and need some tips, correct? Well, I am more than happy to help. It’s pretty easy. I see you have a Windows laptop like me, so you can follow along just fine!

First, let’s start by installing Docker. You will need to follow the instructions specified for your operating system. Make sure your version of Windows is compatible.

Now that you have Docker, you can run Docker commands using the Docker executable.

Because it’s common to start developing with a “Hello World!” program, let’s run a “Hello World!” container. Try running the command docker run busybox echo "Hello world" and you should get a similar output:

> docker run busybox echo "Hello world"
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
90e01955edcd: Pull complete
Digest: sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Status: Downloaded newer image for busybox:latest
Hello world

Allow me to explain what running this command did for us. Docker first searched for the image we are trying to pull on your local machine but couldn’t find it. The image was pulled from Docker Hub instead, Docker’s public registry of ready-made container images. The image we pulled is a BusyBox image, which combines tiny UNIX tools into a single executable. Then, Docker created an isolated container based on the image. Optionally, we specified which command to execute when running the container. We downloaded and ran a full application without the need to install it or any of its dependencies, all in the same command. Fascinating, don’t you agree?

What’s a docker run command?

Now, let me elaborate a bit more on the docker run command. This command runs existing images or pulls images from the Docker Hub registry. These images are software packages that get updated often, so there is more than one version for each image. Docker allows multiple versions of an image with the same name, but each version must have a unique tag. If you run the docker run <image> command without a tag, Docker assumes you are looking for the latest version of the image, which has the latest tag. To specify the version of the image you are looking for, simply add the tag docker run <image>:<tag>.

You might want to list the images using docker images to check the images created, their tags (or versions), creation dates, and sizes. After you run it, you should get an output similar to the following example:

> docker images
REPOSITORY    TAG      IMAGE ID       CREATED       SIZE
busybox       latest   59788edf1f3e   8 weeks ago   1.15MB

You can also use the docker container list command to list the running containers. If you run it right now, you probably won’t get any containers listed because the container no longer running. But if you add the -a or --all flag, both running and stopped containers are displayed in a similar output to this example:

>docker container list -a
CONTAINER ID IMAGE COMMAND CREATED ... 47130c55f730 busybox "echo 'Hello world'" About an hour ago ...

(Some of the details are omitted and replaced by ...)

Do you find the command docker container list a bit long? If so, there is an alternative command, docker ps, with the same function. You can optionally add the -a flag to show the stopped containers as well.

Since the container shows up as a stopped container, you can start it up again by using the docker start <container ID> command. And, you can stop a running container by using the command docker stop <container ID>.

Create a Docker image

Now that you know how to run a new container using an image from the Docker Hub registry, let’s make our own Docker image. The image we will create consists mainly of two things: the application you want to run and the Dockerfile that Docker reads to automatically build an image for our application. The Dockerfile is a document that contains all the commands that Docker users could call on the command line to assemble an image. Let’s first start with the simple Node.js application, and name it app.js. Feel free to customize the name if you’d like.

const http = require('http');
const os = require('os');var server = http.createServer(function(req,res){
  response.end("Hostname is " + os.hostname() + "\n");
})
server.listen(3000);

As you can see in this code sample, we are just starting an HTTP server on port 3000, which will respond with “Hostname is (the hostname of the server host)” to every request. Make a directory and name it as you like, then save the app code inside of it. Make sure no other files are present in that directory.

Now that we’ve created an application, it’s time to create our Dockerfile. Create a file called Dockerfile, copy and paste the content from the following code sample into that file, and then save it in the same directory as your app code.

FROM node:8
COPY app.js /app.js
CMD ["node", "app.js"]

Each FROM statement has a meaning in the Dockerfile. FROM designates which parent image you are using as a base for the image you are building. It is always better to choose a proper base image. We could have written FROM Ubuntu, but using a general-purpose image for running a Node application is unnecessary, because it increases the image overhead. In general, the smaller the better.

Instead, we used the specialized official Node runtime environment as a parent image. Another thing to note is that we specified the version with the tag FROM node:8 instead of using the default latest tag. Why? The latest tag result in a different base image used when a new version is released, and your build may break. I prefer to take this precaution.

We also used COPY <src> <dest> to copy new files or directories from <src> and add them to the file system of the container at the path <dest>. The COPY instruction copies new files or directories from <src> and adds them to the filesystem of the container at the <dest> path. Another Dockerfile instruction that has a similar function as COPY, is ADD. However, COPY is preferred because it is simpler. You can use ADD for some unique functions like downloading external resources or extracting .tar files into the image. You can explore that option further by checking out the Docker documentation.

Lastly, you can use the CMD instruction to run the application contained by your image. The command, in this case, would be node app.js.

There are other instructions that can be included in the Dockerfile. Reviewing them briefly now could prove helpful for you later on. The RUN command, for example, allows you to run commands to set up your application and you can use it to install packages. An example of that is RUN npm install. We can expose a specific port to the world outside the container we are building by using EXPOSE <port>.

Before writing all these commands, you should know some essential knowledge about Dockerfiles. Every command you write in the Dockerfile creates a layer, and each layer is cached and reused. Invalidating the cache of a single layer invalidates all the subsequent layers below it. For example, invalidation occurs after command change. Something to note is that Docker likes to keep the layers immutable. So, if you add a file in one layer and remove it in the next one, the image still contains that file on the first layer. It’s just that now the container doesn’t have access to it anymore.

Two things to keep in mind is that the fewer layers in a Dockerfile, the better. To change the inner layers in Docker images, Docker must remove all the layers above it first. Think about it like this: you’ll cry less if you have fewer layers to peel off an onion. Also, the most general steps and the longest steps should come first in your Dockerfile (the inner layers), while the specific ones should come later (outer layers).

Build an image from a Dockerfile

Now that you have a better understanding of the contents of the Dockerfile, let’s go ahead and build an image. First, make sure your path is inside the new directory that you made. Using ls command should only show you two files: app.js and Dockerfile. You can build the image by using the docker build -t medium. command. We tag it medium by using the -t flag and we target the current directory. (Note the dot at the end of the following command.)

>docker build -t medium .
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM node:8
8: Pulling from library/node
54f7e8ac135a: Pull complete
d6341e30912f: Pull complete
087a57faf949: Pull complete
5d71636fb824: Pull complete
0c1db9598990: Pull complete
89669bc2deb2: Pull complete
3b96ee2ed0b3: Pull complete
df3df33f8e3c: Pull complete
Digest: sha256:dd2381fe1f68df03a058094097886cd96b24a47724ff5a588b90921f13e875b7
Status: Downloaded newer image for node:8
---> 3b7ecd51ffe5
Step 2/3 : COPY app.js /app.js
---> 63633b2cf6e7
Step 3/3 : CMD ["node", "app.js"]
---> Running in 9ced576fdb46
Removing intermediate container 9ced576fdb46
---> 91c37fa82fe5
Successfully built 91c37fa82fe5
Successfully tagged medium:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

You can use the run command again to run the built image, or you can use docker push to push it to a registry and pull it again on another computer from the registry using docker pull.

Congratulations! Now you know how to make a proper Dockerfile, build an image from that Dockerfile, and run it. These skills will help you send young Appy out into the world. If you’re feeling adventurous, you can check out the Docker docs and keep going. Good luck!



A previous version of this post was published on Medium.

Amro Moustafa

I shellcheck and you should too

This post is about ShellCheck and the power that can come from using it to find bugs in shell scripts. Over the years, I wrote a lot of code that glues things together, as a DevOps engineer, production engineer, and developer advocate. Since I spend most of my time on Linux and Mac, bash is the obvious choice for me. However, I started talking to people at different conferences and found out that the idea of linting code is still considered a developer practice, instead of an operator practice, so I’m here to try to set the record straight.

Why

First, let me answer why I think linting code is an operator practice and you should use ShellCheck to do it. Standardizing on coding practices (yes, you have to code) will save you from insane overhead costs and tech debt in the future. When your team’s bash scripts start to look the same, they are easier to read, allowing people to start paying attention to what’s happening instead of where the do command or semicolon is. Running shellcheck enforces this. I should say that you can override ShellCheck suggestions, but that’s a team decision and out of the scope of this article. (Read up about ignoring a ShellCheck error if you need to go down this path.)

How

Now, let’s talk about how to use ShellCheck. There are a handful of ways to get it: apt, yum, homebrew, and even docker can be used to run this application. It’s also easy to add it to your continuous integration and continuous delivery (CI/CD) pipeline by pulling out every .sh file and running shellcheck against it.

Even adding it to your Makefile is simple:

check-scripts:
    # Fail if any of these files have warnings
    shellcheck myscripts/*.sh

Even Travis CI has ShellCheck built-in. Just add the following to your .travis.yml:

script:
  # Fail if any of these files have warnings
  - shellcheck myscripts/*.sh

Personally, I use a Docker container on my local laptop. So, I wrote a simple script named shellcheck.sh and any time I save a script .sh, I run shellcheck.sh script.sh against it. As you can see, it’s very straight forward:

#!/bin/bash

if [ $# -eq 0 ]
  then
    echo "You need to add an script to this check, shellcheck.sh script.sh"
    exit 1
else
  if [[ "$(ping www.google.com -c 1 | grep '100% packet loss' )" != "" ]]
  Then
    echo "You can't seem to connect to the internet, you should probably fix that."
    exit 1
  fi
  docker pull koalaman/shellcheck:latest
  docker run -v "$PWD:/mnt" koalaman/shellcheck "$1"
fi

Knowing my code is standardized and works as expected gives me peace of mind. The output of shellcheck shows exactly what your offending code is and there is a dedicated wiki page explaining why you should not code that way. I applaud the main developer, Vidar Holen, and his team for helping us become better developers.

Summary

Honestly, I’m just here to convince you, dear reader, to give this linter a shot. The overhead expense is minimal, and the benefits are immense. This is one of those things that you shouldn’t waste time debating, since it will make your life, and your team’s life, easier. You can integrate it into CI, run locally, and make sure everyone writes their scripts with a unified pattern for only a tad bit of work.

JJ Asghar