k8s-ocp-yaml
Search…
2019-07-02-openshift origin 3.11 在线安装
采用单节点模式,在centos7.4安装openshift3.11社区版,环境需联网。 若离线安装,提前拉好源和镜像。

配置环境

1. 系统配置

操作系统: centos 7.4 mini install,最小化安装 虚机配置: 2核 8G 网卡配置IP netmask gateway dns

2. 设置主机名及配置ssh免密

1
# 设置主机名
2
hostnamectl set-hostname origin311.localpd.com
3
4
# /etc/hosts 增加一行,把IP换成自己虚机的
5
172.16.160.13 origin311.localpd.com
6
7
# 配置ssh免密
8
ssh-keygen # 一路敲回车
9
ssh-copy-id origin311.localpd.com # 输入密码
10
ssh origin311.localpd.com hostname -i # 验证免密
Copied!

3. 更新yum源及设置selinux

注意此处不要用官方的epel源,那个ansible是2.8.0的,按照官方文档,ansible版本需不低于2.5.7,不支持2.8.0,此处自己添加ansible源,安装2.7.4
1
# 添加ansible 源
2
cat > /etc/yum.repos.d/ansible.repo << EOF
3
[ansible]
4
name=ansible
5
baseurl=https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/
6
enabled=0
7
gpgcheck=0
8
EOF
9
10
# 安装 ansible
11
yum -y --enablerepo=ansible install ansible-2.7.4
12
13
# 安装其他需要的包及更新
14
yum install pyOpenSSL wget git net-tools bind-utils yum-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct java-1.8.0-openjdk-headless python-passlib -y
15
16
yum update -y
17
18
# 安装并启动docker
19
yum install docker-1.13.1 -y
20
systemctl enable docker
21
systemctl start docker
22
systemctl is-active docker
23
24
# 修改selinux
25
setenforce 0
26
27
# 关闭防火墙和设置selinux 为permissive
28
systemctl disable firewalld;
29
systemctl stop firewalld;
30
31
# /etc/selinux/config 把SELINUX=enforcing 改为 SELINUX=permissive
32
sed -i -e "s/^SELINUX=enforcing/SELINUX=permissive/" /etc/selinux/config
33
34
reboot # 必须做
Copied!

docker安装配置

1. docker 安装

yum install docker-1.13.1
Verify that version 1.13 was installed:
1
rpm -V docker-1.13.1
2
docker version
Copied!

2. docker 独立存储配置,需要独立sdb磁盘,POC和测试环境忽略这步

停止docker服务
1
systemctl stop docker
2
rm -rf /var/lib/docker/
Copied!
配置/etc/sysconfig/docker-storage-setup
1
cat <<EOF > /etc/sysconfig/docker-storage-setup
2
STORAGE_DRIVER=overlay2
3
DEVS=/dev/sdb
4
CONTAINER_ROOT_LV_NAME=dockerlv
5
CONTAINER_ROOT_LV_SIZE=100%FREE
6
# 下面这个mount的路径就比较特殊了,其实这个相关的配置文件就是使用sdb创建一个pv、vg、lv,然后格式化成xfs,最后mount一下,docker只是去使用这个路径下空间,如果docker存储的路径需要改变,这个mount的路径也需要进行修改。
7
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
8
VG=docker-vg
9
EOF
Copied!
初始化docker存储启动docker
1
docker-storage-setup
2
systemctl start docker
Copied!
最后通过docker info去查看存储相关的配置进行验证
注意点 如果此前这块盘被使用过,创建了pv,vg,lv,需依次删除lv、vg、pv,然后再使用fdisk删除sdb1分区,执行partprobe,最后再执行docker-storage-setup。 如果pv删了,vg删不掉,重启下系统。 在使用一块盘、分区、VG,lv默认使用的是所在VG的40%空间,这个要看具体 执行docker-storage-setup报错如下错误时,将WIPE_SIGNATURES=true追加至/etc/sysconfig/docker-storage-setup文件后。
ERROR: Found dos signature on device /dev/sdb at offset 0x1fe. Wipe signatures using wipefs or use WIPE_SIGNATURES=true and retry.
如果再次执行docker-storage-setup还会报如下错,删除/etc/sysconfig/docker-storage文件再次执行即可。
docker-storage-setup

3. 启动docker服务

1
systemctl enable docker
2
systemctl start docker
3
systemctl is-active docker
Copied!
修改docker配置 /etc/sysconfig/docker 替换之前的OPTIONS
1
cp /etc/sysconfig/docker /etc/sysconfig/docker.bak.$(date "+%Y%m%d%H%M%S");
2
sed -i s/".*OPTIONS=.*"/"OPTIONS='--log-driver=json-file --insecure-registry=172.30.0.0\/16 --insecure-registry=registry.ocp311origin.com:5000 --selinux-enabled --log-opt max-size=1M --log-opt max-file=3'"/g /etc/sysconfig/docker;
3
4
##Restart the Docker service:
5
systemctl restart docker
Copied!

openshift安装

1. 下载ansible 脚本

1
cd ~
2
git clone https://github.com/openshift/openshift-ansible
3
cd openshift-ansible
4
git checkout release-3.11
5
6
# 也可以手动去网站下载release-3.11的分支
Copied!

2. 准备ansible hosts文件

1
# 在openshift-ansible 目录
2
# 新建文件 inventory/hosts 内容如下
3
4
[[email protected] openshift-ansible-release-3.11]# cat inventory/hosts
5
# Create an OSEv3 group that contains the masters, nodes, and etcd groups
6
[OSEv3:children]
7
masters
8
nodes
9
etcd
10
11
# Set variables common for all OSEv3 hosts
12
[OSEv3:vars]
13
# SSH user, this user should allow ssh based auth without requiring a password
14
ansible_ssh_user=root
15
16
# If ansible_ssh_user is not root, ansible_become must be set to true
17
#ansible_become=true
18
19
openshift_deployment_type=origin #代表开源版
20
21
# 指定安裝的 OpenShift 版本
22
openshift_release="3.11"
23
openshift_image_tag=v3.11.0
24
openshift_pkg_version=-3.11.0
25
openshift_use_openshift_sdn=true
26
27
os_sdn_network_plugin_name='redhat/openshift-ovs-networkpolicy'
28
# When installing osm_cluster_network_cidr and openshift_portal_net must be set.
29
# Sane examples are provided below.
30
#osm_cluster_network_cidr=10.128.0.0/14
31
#openshift_portal_net=172.30.0.0/16
32
33
# disable checks unsupported
34
openshift_disable_check=docker_storage,memory_availability,docker_image_availability,disk_availability,docker_storage_driver
35
36
# uncomment the following to enable htpasswd authentication; defaults to AllowAllPasswordIdentityProvider
37
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
38
# Defining htpasswd users
39
#openshift_master_htpasswd_users={'user1': '<pre-hashed password>', 'user2': '<pre-hashed password>'}
40
41
# default subdomain to use for exposed routes, you should have wildcard dns
42
# for *.apps.test.example.com that points at your infra nodes which will run
43
# your router
44
openshift_master_default_subdomain=apps.localpd.com
45
46
#Set cluster_hostname to point at your load balancer
47
#将来平台的访问域名
48
openshift_master_cluster_method=native
49
openshift_master_cluster_hostname=origin311.localpd.com
50
openshift_master_cluster_public_hostname=origin311.localpd.com
51
52
# Cluster metrics are not set to automatically deploy.
53
# The metrics public URL can be set during cluster installation using
54
# the openshift_metrics_hawkular_hostname Ansible variable, which defaults to:
55
# https://hawkular-metrics.{{openshift_master_default_subdomain}}/hawkular/metrics
56
openshift_metrics_install_metrics=true
57
58
ansible_service_broker_install=false
59
openshift_enable_service_catalog=false
60
template_service_broker_install=false
61
openshift_logging_install_logging=false
62
enable_excluders=false
63
64
# registry passwd
65
#oreg_url=registry.ocp311origin.com:5000/openshift3/ose-${component}:${version}
66
#oreg_url=registry.ocp311origin.com:5000/openshift/origin-${component}:${version}
67
openshift_examples_modify_imagestreams=true
68
69
# Enable cockpit
70
osm_use_cockpit=true
71
#
72
# Set cockpit plugins
73
osm_cockpit_plugins=['cockpit-kubernetes']
74
75
# docker config
76
#openshift_docker_additional_registries=registry.ocp311origin.com:5000
77
openshift_docker_insecure_registries=registry.ocp311origin.com:5000
78
#openshift_docker_blocked_registries
79
openshift_docker_options="--insecure-registry 172.30.0.0/16 --log-driver json-file --log-opt max-size=1M --log-opt max-file=3"
80
81
# OpenShift Router Options
82
# Router selector (optional)
83
# Router will only be created if nodes matching this label are present.
84
# Default value: 'node-role.kubernetes.io/infra=true'
85
#openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'
86
#
87
# Router replicas (optional)
88
# Unless specified, openshift-ansible will calculate the replica count
89
# based on the number of nodes matching the openshift router selector.
90
#openshift_hosted_router_replicas=2
91
92
# Openshift Registry Options
93
# Registry selector (optional)
94
# Registry will only be created if nodes matching this label are present.
95
# Default value: 'node-role.kubernetes.io/infra=true'
96
#openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'
97
#
98
# Registry replicas (optional)
99
# Unless specified, openshift-ansible will calculate the replica count
100
# based on the number of nodes matching the openshift registry selector.
101
#openshift_hosted_registry_replicas=2
102
103
104
# openshift_cluster_monitoring_operator_install=false
105
# openshift_metrics_install_metrics=true
106
# openshift_enable_unsupported_configurations=True
107
#openshift_logging_es_nodeselector='node-role.kubernetes.io/infra: "true"'
108
#openshift_logging_kibana_nodeselector='node-role.kubernetes.io/infra: "true"'
109
110
111
# host group for masters
112
[masters]
113
origin311.localpd.com
114
115
# host group for etcd
116
[etcd]
117
origin311.localpd.com
118
119
# host group for nodes, includes region info
120
[nodes]
121
origin311.localpd.com openshift_node_group_name='node-config-all-in-one'
Copied!

3. 执行安装脚本

1
# 使用ansible脚本预检查
2
ansible-playbook -i inventory/hosts playbooks/prerequisites.yml
3
4
# 执行完成会在每台节点添加openshift在线源,此源下载比较慢。若使用离线源,可以手动替换。 /etc/yum.repos.d/CentOS-OpenShift-Origin311.repo
5
6
# 执行部署脚本
7
ansible-playbook -i inventory/hosts playbooks/deploy_cluster.yml
8
9
# 卸载脚本,如果执行出错或者想要重新部署可以执行卸载脚本
10
正常情况下不要执行
11
### ansible-playbook -i inventory/hosts playbooks/adhoc/uninstall.yml ###
Copied!
确认部署结果
1
oc get node
2
oc get pod --all-namespaces
3
4
// 设置密码,用户名密码都是 admin
5
htpasswd -b /etc/origin/master/htpasswd admin admin; #这个命令每个master节点都要做
6
oc adm policy add-cluster-role-to-user cluster-admin admin; #master1 执行即可
7
8
// 如果有其他用户,多次执行,再通过oc admin赋权
9
htpasswd -b /etc/origin/master/htpasswd <usre_name> <user_passwd>
Copied!

4. 部署FAQ,异常处理

  1. 1.
    部署报错 Install docker excluder - yum
TASK [openshift_excluder : Install docker excluder - yum] fatal: [node2.ocp311origin.com]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure talking to yum: 'ascii' codec can't encode characters in position 173-177: ordinal not in range(128)"}
这个报错,是因为脚本部署过程修改了/etc/resolv.conf 中nameserver,改成了本地IP,而dnsmasq未正常启动,不能访问外部域名,nameserver 被改成本地IP,这个是正常的,只要dnsmasq服务正常,就没有问题,之前的dns配置文件被移到了 /etc/dnsmasq.d/origin-upstream-dns.conf 查看dnsmasq服务发现报错信息,手动启动dnsmasq起不来,和dbus有关,可能是部署过程更新了dbus导致异常。
解决方法:重启dbus,然后重启dnsmasq并查看dnsmasq状态,还不行就重启操作系统。
  1. 1.
    部署报错 Wait for control plane pods to appear
TASK [openshift_control_plane : Wait for control plane pods to appear] ** Wednesday 29 May 2019 16:09:27 +0800 (0:00:00.098) 0:06:23.053 * FAILED - RETRYING: Wait for control plane pods to appear (60 retries left).
ansible版本问题,2.8.0 版本会卡在这地方,2.7.4正常,在github上官方有说明,不低于2.5.7,并且低于2.8.0 解决方法: 更换ansible版本,如果严格执行上述步骤,ansible版本应该是正确的,不会出现此问题  部署停留在Verify that the console is running
  1. 1.
    TASK [openshift_web_console : Verify that the console is running] *
    Thursday 30 May 2019 14:04:12 +0800 (0:00:00.109) 0:04:22.223 **
    1. 1.
      Hosts: origin311.localpd.com
      Play: OpenShift Metrics
      Task: openshift_metrics : fail
      Message: 'keytool' is unavailable. Please install java-1.8.0-openjdk-headless on the control node
      Failure summary:
  2. 2.
    Hosts: origin311.localpd.com
    Play: OpenShift Metrics
    Task: generate htpasswd file for hawkular metrics
    Message: This module requires the passlib Python library
解决方法: 因为装metrics需要 yum install java-1.8.0-openjdk-headless python-passlib