by ·0 Comments

Recently played with the Spring/SpringBoot/SpringCloud stack with a toy project: https://github.com/gonwan/spring-cloud-demo. Just paste README.md here, and any pull request is welcome:

Introduction

The demo project is initialized from https://github.com/carnellj/spmia-chapter10. Additions are:

  • Code cleanup, bug fix, and better comments.
  • Java 9+ support.
  • Spring Boot 2.0 migration.
  • Switch from Postgres to MySQL, and from Kafka to RabbitMQ.
  • Easier local debugging by switching off service discovery and remote config file lookup.
  • Kubernetes support.
  • Swagger Integration.
  • Spring Boot Admin Integration.

The project includes:

  • [eureka-server]: Service for service discovery. Registered services are shown on its web frontend, running at 8761 port.
  • [config-server]: Service for config file management. Config files can be accessed via: http://${config-server}:8888/${appname}/${profile}. Where ${appname} is spring.application.name and ${profile} is something like dev, prd or default.
  • [zipkin-server]: Service to aggregate distributed tracing data, working with spring-cloud-sleuth. It runs at 9411 port. All cross service requests, message bus delivery are traced by default.
  • [zuul-server]: Gateway service to route requests, running at 5555 port.
  • [authentication-service]: OAuth2 enabled authentication service running at 8901. Redis is used for token cache. JWT support is also included. Spring Cloud Security 2.0 saves a lot when building this kind of services.
  • [organization-service]: Application service holding organization information, running at 8085. It also acts as an OAuth2 client to authentication-service for authorization.
  • [license-service]: Application service holding license information, running at 8080. It also acts as an OAuth2 client to authentication-service for authorization.
  • [config]: Config files hosted to be accessed by config-server.
  • [docker]: Docker compose support.
  • [kubernetes]: Kubernetes support.

NOTE: The new OAuth2 support in Spring is actively being developed. All functions are merging into core Spring Security 5. As a result, current implementation is suppose to change. See:

Tested Dependencies

  • Java 8+
  • Docker 1.13+
  • Kubernetes 1.11+

Building Docker Images

In case of running out of disk space, clean up unused images and volumes with:

Running Docker Compose

Or with separate services:

Running Kubernetes

NOTE: Kubernetes does not support environment variable substitution by default.

Use Cases

Suppose you are using the kubernetes deployment.

Get OAuth2 token

curl is used here, and 31004 is the cluster-wide port of the Zuul gateway server:

Get organization info

Use the token returned from previous request.

Get license info associated with organization info

Use the token returned from previous request.

Distributed Tracing via Zipkin

Every response contains a correlation ID to help diagnose possible failures among service call. Run with curl -v to get it:

Search it in Zipkin to get all trace info, including latencies if you are interested in.
zipkin-1
zipkin-2

The license service caches organization info in Redis, prefixed with organizations:. So you may want to clear them to get a complete tracing of cross service invoke.

Working with OAuth2

All OAuth2 tokens are cached in Redis, prefixed with oauth2:. There is also JWT token support. Comment/Uncomment @Configuration in AuthorizationServerConfiguration and JwtAuthorizationServerConfiguration classes to switch it on/off.

Swagger Integration

The organization service and license service have Swagger integration. Access via /swagger-ui.html.

Spring Boot Admin Integration

Spring Boot Admin is integrated into the eureka server. Access via: http://${eureka-server}:8761/admin.
sba-1

by ·0 Comments

It is painful to deploying a Kubernetes cluster in mainland China. The installation requires access to Google servers, which is not so easy for every one. Fortunately, there are mirrors or alternative ways. I’ll use Docker v1.13 and Kubernetes v1.11 in the article.

1. Install Docker

CentOS SCL should be enabled first.

2. Install Kubernetes

2.1 Add the Aliyun mirror for Kubernetes packages

2.2 Precheck OS environmemt

Run the init command by specify the version, the access to Google server is avoided. The script also advices you to turn off firewalld, swap, selinux and enable kernel parameters:

Open /etc/sysconfig/selinux, change enforcing to permissive.
Create /etc/sysctl.d/k8s.conf with content:

Remember to comment out swap volumes from /etc/fstab.

2.3 Pull Kubernates images

Pull the Kubernetes images from docker/docker-cn mirror maintained by anjia0532. These are minimal images required for a Kubernetes master installation.

These version numbers comes from the kubeadm init command if you cannot access Google servers. These images should be retagged to gcr.io ones before next steps, or the kubeadm command line would not find them:

Now the output of docker images looks like:

Also KUBE_REPO_PREFIX and other environment variables can be used to customize the prefix. I have no time to verify them.

2.4 Start the Kubernetes master

Run the init script again and it will success with further guidelines:

Run the mkdir/cp/chown command to enable kubectl usage. Then add the weave pod network. It may take some time, since images are pulled.

Now the master is finished, verify with the Ready status:

2.4 Start the Kubernetes node(slave)

A Kubernetes node only requires kube-proxy-amd64 and pause images, pull these ones:

Weave images can also been prefetched:

Join the node to our Kubernetes master by running the command line in the kubeadm init output:

3. Verify Kubernetes cluster status

Verify nodes with:

Verify internal pods with:

If the status of a pod is not Running, get the detailed info from:

If something goes wrong, and you cannot restore from it, simply reset the master/node:

4. Install Kubernetes Dashboard

By default, all user pods are allocated on Kubernetes nodes(slaves). Pull the dashboard image in advance on the node machine:

Install with alternative setup, since recommended setup is not so friendly in a development envronment:

Refer here for remote access:

Change type: ClusterIP to type: NodePort and save file. Next we need to check port on which Dashboard was exposed.

Now, you can access with: http://<master-ip>:31023/.
You can grant admin grant full admin privileges to Dashboard’s Service Account in the development environment for convenience:

5. Troubleshoting

In my office environment, errors occur and the coredns are always in CrashLoopBackOff status:

I Googled a lot, read answers from Stackoverflow and Github, reset iptables/docker/kubernetes, but still failed to solve it. There ARE unresolved issues like #60315. So I tried to switch to flannel network instead of weave. First, Kubernetes and weave need to be reset:

This time, initialize kubeadm and network with:

The flannel image can be pulled first:

Everything works. Also referred here.

Updated May 7, 2019: Kubernetes 1.13 finally add a command line switch to use an alternative repository. Simply run kubeadm with:

And verify with docker images.

Updated May 10, 2019: If using Ubuntu/Linuxmint, add repository with:

Updated June 3, 2019: flannel seems to have a close version dependency on kubernetes version. When deploying kubernetes 1.14, a specific git version should be used, according to the official document:

Updated Jan 11, 2022: Just deployed a new cluster with docker 20.10.12 & kubernetes 1.23.1.
1. kubeadm defaults to systemd, instead of cgroupfs as the container runtime cgroup driver. In docker case, edit /etc/docker/daemon.json, and restart docker service:

2. flannel script updated:

3. kubernetes dashboard script updated:

The recommended configuration enables HTTPS, and an auto-generated certificate is used. Now follow the document to create an admin user and get the login token: https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md. Get the token with: