2. 多节点部署

前面我们已经部署了单节点的OpenStack环境,这次我们尝试部署一个多节点的环境(包括1个controller节点,1个compute节点,1个storage节点),之前的单节点因为已经安装了kolla-ansible和docker registry,我们把它当做部署节点。

环境准备

准备三台服务器,分别作为controller节点,compute节点,storage节点。其中controller节点3张网卡,compute、storage节点2张网卡。操作系统为centos7.4

1)关闭Selinux

vi /etc/sysconfig/selinux
SELINUX=disabled

2)关闭firewalld

systemctl stop firewalld
systemctl disable firewalld

3)设置主机名,hosts文件

vi /etc/hosts
172.16.31.130 control01
172.16.31.131 controller
172.16.31.132 compute
172.16.31.133 storage

4)storage节点加盘

为storage节点划分磁盘,作为cinder的lvm后端,storage节点有13tb的硬盘采用parted分区,分区过程略去

k-17

创建pv、vg

[root@storage ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
[root@storage ~]# vgcreate cinder-volumes /dev/sdb
Volume group "cinder-volumes" successfully created

修改配置

vi /etc/lvm/lvm.conf
k-18

重启lvm服务

systemctl restart lvm2-lvmetad.service

5)确认节点配置

controller 3 张网卡

compute 2 张网卡

storage 2 张网卡(其实一张即可,因为不需要tunnel网络)

storage 磁盘已经分区

k-19

安装docker

1)下载相关docker rpm包

这里我已经到docker把docker的rpm下载下来了,我们上传的相关的目录下,当然还是需要配centos和epel的源才能正常安装的

k-20

2)安装Docker 1.12.6(三台服务器都要安装)

yum install docker-engine-* -y

在各个节点安装

yum install python-docker-py -y

3)设置Docker

mkdir /etc/systemd/system/docker.service.d
tee /etc/systemd/system/docker.service.d/kolla.conf << 'EOF'
[Service]
MountFlags=shared
EOF

4)重启相关服务

systemctl daemon-reload
systemctl enable docker
systemctl restart docker

5)编辑/usr/lib/systemd/system/docker.service文件

ExecStart=/usr/bin/dockerd --insecure-registry 172.16.31.130:4000

6)重启Docker服务

systemctl daemon-reload
systemctl restart docker

配置免密登录

部署节点和其他节点免密登陆

[root@control01 ~]# ssh-keygen
[root@control01 ~]# ssh-copy-id controller
[root@control01 ~]# ssh-copy-id compute
[root@control01 ~]# ssh-copy-id storage

部署OpenStack

1)停止部署节点容器

因为部署节点之前已经部署了All in One的OpenStack,为了降低资源占用,我们可以先停止本节的OpenStack相关容器。

[root@control01 ~]# cd /root/kolla-ansible-4.0.3.dev36/tools/
[root@control01 tools]# . kolla-ansible stop

2)修改multinode文件

[root@control01 ~]# cd /root/kolla-ansible-4.0.3.dev36/
[root@control01 kolla-ansible-4.0.3.dev36]# cp ansible/inventory/* /home/
[root@control01 ~]# cd /home
[root@control01 home]# ls
all-in-one multinode
[root@control01 home]# cp multinode mymultinode
[root@control01 home]# vi mymultinode
[control]
controller
[network]
controller
[compute]
compute
[monitoring]
controller
[storage]
storage
......

里面的内容很好理解,表示一个控制节点 controller,网络节点也安装到控制节点,一个计算节点 compute ,一个存储节点 storage,后面的部分不用修改

3)修改global和password配置文件

[root@control01 ~]# cd /etc/kolla/
[root@control01 kolla]# vi globals.yml

————————————分割线———————————————

kolla_internal_vip_address: "172.16.31.131" 外部访问地址,如果是非HA环境,是控制节点IP
docker_registry: "172.16.31.130:4000" 镜像库,用部署节点的镜像库
docker_namespace: "99cloud"
network_interface: "eno1" 网络接口,默认就这一个接口,这里我们以eth0作为API网络
api_interface: "{{ network_interface }}" 用的是eno2
tunnel_interface: "eno2" eno2作为vxlan的承载网络
neutron_external_interface: "eno3" eno3作为外部网络
enable_cinder: "yes" 启用cinder
enable_cinder_backend_lvm: "yes" cinder后端用lvm
cinder_volume_group: "cinder-volumes"
enable_haproxy: "no" 不启用haproxy

————————————分割线———————————————

[root@control01 kolla]# vi passwords.yml
keystone_admin_password: admin admin用户登录密码

4)部署前检查

[root@control01 ~]# cd /root/kolla-ansible-4.0.3.dev36/tools/
[root@control01 ~]# . kolla-ansible prechecks -i /home/mymultinode

没有报错即可正常部署

5)部署

[root@control01 tools]# . kolla-ansible deploy -i /home/mymultinode

开始部署,等待大约20分钟,部署完成,一般不会有什么问题,如果有问题,则需要根据报错及容器日志查找原因

k-21

登录dashboard http://172.16.31.131

k-22

登录系统,查看各个服务正常,表明部署成功。

可以通过docker ps命令查看各个节点运行的容器

controller节点:

[root@controller ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ecab1b4246e 172.16.31.130:4000/99cloud/centos-source-horizon:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours horizon
b264ea5467ff 172.16.31.130:4000/99cloud/centos-source-heat-engine:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours heat_engine
08ab0aa8603c 172.16.31.130:4000/99cloud/centos-source-heat-api-cfn:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours heat_api_cfn
6f6aba4804a2 172.16.31.130:4000/99cloud/centos-source-heat-api:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours heat_api
67729f946b23 172.16.31.130:4000/99cloud/centos-source-neutron-metadata-agent:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_metadata_agent
f8f8d9f766e9 172.16.31.130:4000/99cloud/centos-source-neutron-l3-agent:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_l3_agent
fd3a8ca63fdf 172.16.31.130:4000/99cloud/centos-source-neutron-dhcp-agent:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_dhcp_agent
c9f9f72090a4 172.16.31.130:4000/99cloud/centos-source-neutron-openvswitch-agent:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_openvswitch_agent
9f19bca0759c 172.16.31.130:4000/99cloud/centos-source-neutron-server:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_server
e1286083b3f7 172.16.31.130:4000/99cloud/centos-source-openvswitch-vswitchd:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours openvswitch_vswitchd
dd4986364575 172.16.31.130:4000/99cloud/centos-source-openvswitch-db-server:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours openvswitch_db
a6692acea93f 172.16.31.130:4000/99cloud/centos-source-nova-novncproxy:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_novncproxy
cd704f0bc2c7 172.16.31.130:4000/99cloud/centos-source-nova-consoleauth:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_consoleauth
105978eda1a9 172.16.31.130:4000/99cloud/centos-source-nova-conductor:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_conductor
9d9e7571d7dd 172.16.31.130:4000/99cloud/centos-source-nova-scheduler:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_scheduler
7436ca003817 172.16.31.130:4000/99cloud/centos-source-nova-api:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_api
ae8d830091d3 172.16.31.130:4000/99cloud/centos-source-nova-placement-api:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours placement_api
f053087cfde5 172.16.31.130:4000/99cloud/centos-source-cinder-scheduler:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cinder_scheduler
8c849182ee63 172.16.31.130:4000/99cloud/centos-source-cinder-api:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cinder_api
129996bd76ee 172.16.31.130:4000/99cloud/centos-source-glance-registry:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours glance_registry
fcc184936be5 172.16.31.130:4000/99cloud/centos-source-glance-api:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours glance_api
06a35ba7eb7f 172.16.31.130:4000/99cloud/centos-source-keystone:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours keystone
6821886162bc 172.16.31.130:4000/99cloud/centos-source-rabbitmq:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours rabbitmq
04b556da8365 172.16.31.130:4000/99cloud/centos-source-mariadb:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours mariadb
859a1d8d0137 172.16.31.130:4000/99cloud/centos-source-cron:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cron
4e7230af5339 172.16.31.130:4000/99cloud/centos-source-kolla-toolbox:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours kolla_toolbox
50be20c91704 172.16.31.130:4000/99cloud/centos-source-fluentd:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours fluentd
72b2538e1beb 172.16.31.130:4000/99cloud/centos-source-memcached:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours memcached

compute节点:

[root@compute ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0f954ef9095e 172.16.31.130:4000/99cloud/centos-source-neutron-openvswitch-agent:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours neutron_openvswitch_agent
8f62c376e2b5 172.16.31.130:4000/99cloud/centos-source-openvswitch-vswitchd:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours openvswitch_vswitchd
005c3dd582a9 172.16.31.130:4000/99cloud/centos-source-openvswitch-db-server:4.0.2.1 "kolla_start" 24 hours ago Up 24 hours openvswitch_db
d6a9b676c0ad 172.16.31.130:4000/99cloud/centos-source-nova-compute:4.0.2.1 "kolla_start" 36 hours ago Up 36 hours nova_compute
fbc66c3f40ba 172.16.31.130:4000/99cloud/centos-source-nova-libvirt:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours nova_libvirt
6f942f376c83 172.16.31.130:4000/99cloud/centos-source-nova-ssh:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours nova_ssh
4872cfb2b025 172.16.31.130:4000/99cloud/centos-source-iscsid:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours iscsid
c55ec16b0752 172.16.31.130:4000/99cloud/centos-source-cron:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cron
426d77add095 172.16.31.130:4000/99cloud/centos-source-kolla-toolbox:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours kolla_toolbox
5f959b2372f0 172.16.31.130:4000/99cloud/centos-sourc

stroage节点:

[root@cinder ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7eca045eeff1 172.16.31.130:4000/99cloud/centos-source-cinder-backup:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cinder_backup
bcdf196510f2 172.16.31.130:4000/99cloud/centos-source-cinder-volume:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cinder_volume
d59fc8ec7716 172.16.31.130:4000/99cloud/centos-source-tgtd:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours tgtd
7a8d9dab2755 172.16.31.130:4000/99cloud/centos-source-iscsid:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours iscsid
ea471af172de 172.16.31.130:4000/99cloud/centos-source-cron:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours cron
4b41ebf3ea62 172.16.31.130:4000/99cloud/centos-source-kolla-toolbox:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours kolla_toolbox
91c086f63421 172.16.31.130:4000/99cloud/centos-source-fluentd:4.0.2.1 "kolla_start" 37 hours ago Up 37 hours fluentd