lost and found ( for me ? )

Deploy OpenStack multi nodes with kolla ( OpenStack in Docker )

Reference
https://greatbsky.github.io/kolla-for-openstack-in-docker/en.html
https://docs.openstack.org/kolla-ansible/latest/quickstart.html

Here are logs when I set up OpenStack multi nodes with kolla.

Build docker images from source, push images to docker private registry, deploy OpenStack with kolla-ansible.

Prepare four nodes

one for docker private registry
one for controller node.
the other two are for compute

All nodes are based on Ubuntu 16.04.

[ network ]

all nodes are belong to 192.168.100.0/24

- hostname : docker-reg

one NIC.
eth0 : 192.168.100.10

- hostname : controller

two NICs

ens3 : 192.168.100.11
ens7 : interface is up, but no ip address

root@controller:~# grep -v ^# /etc/network/interfaces.d/50-cloud-init.cfg
auto lo
iface lo inet loopback

auto ens3
iface ens3 inet static
address 192.168.100.11
netmask 255.255.255.0
gateway 192.168.100.1
dns-nameservers 8.8.8.8 8.8.4.4

auto ens7
iface ens7 inet manual

- hostname : node01

tow NICs

ens3 : 192.168.100.12
ens7 : interface is up, but no ip address

- hostname : node02

two NICs

ens3 : 192.168.100.13
ens7 : interface is up, but no ip address

all nodes have a network reachability with hostname. ( edit /etc/hosts )

[ set up docker private registry ]

install docker and run docker private registry.
root@docker-reg:~# docker run -d -p 5000:5000 --restart=always -v /data/Iaas/Docker/Volumes/Registry:/var/lib/registry --na
me registry registry:2

root@docker-reg:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                    NAMES
97cd11afc792        registry:2          "/entrypoint.sh /etc/"   About a minute ago   Up About a minute   0.0.0.0:5000->5000/tcp   registry

[ enable password-less ssh access among nodes (controller, node01, node02)]

on the controller
root@controller:~# ssh-keygen
root@controller:~# ssh-copy-id -i .ssh/id_rsa.pub root@controller
root@controller:~# ssh-copy-id -i .ssh/id_rsa.pub root@node01
root@controller:~# ssh-copy-id -i .ssh/id_rsa.pub root@node02

do the same thing on both node01 and node02.

[ install docker on controller, node01, node02 ]

root@controller:~# curl -ksSL https://get.docker.io | bash
root@node01:~# curl -ksSL https://get.docker.io | bash
root@node02:~# curl -ksSL https://get.docker.io | bash

[ uninstall lxd on controller, node01, node02 ]

# apt purge lxd -y

[ edit systemd config for docker. ( controller, node01, node02 ) ]

root@controller:~# systemctl edif --full docker

[Service]
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry 192.168.100.10:5000
MountFlags=shared

restart docker
root@controller:~# systemctl restart docker

[ install ntp on controller, node01, node02 ]

root@node02:~# apt install ntp -y

[ install ansible, kolla-* on controller node ]

root@controller:~# apt install python-pip -y
root@controller:~# pip install ansible
root@controller:~# apt-get install python-dev libffi-dev gcc libssl-dev python-selinux
root@controller:~# pip install kolla-ansible

# Note that you also need to install docker-py on node01, and node02.
# do the following on node01 and node02.
# apt install python-pip, pip install docker-py

[ disable libvirt on controller, node01, node02 if it’s running ]

  25  systemctl stop libvirt-bin
  26  systemctl disable libvirt-bin

[ install docker-py on node01, node02 ]

# apt install python-pip
# pip install docker-py

[ build docker images on the controller node and push images to the private registry ]

root@controller:~# cp -r /usr/local/share/kolla-ansible/etc_examples/kolla /etc/kolla/
root@controller:~# cp /usr/local/share/kolla-ansible/ansible/inventory/* .
root@controller:~# cp /etc/kolla/globals.yml /etc/kolla/globals.yml.orig
root@controller:~# cp multinode multinode.orig
root@controller:~# ls
all-in-one  multinode  multinode.orig

edit globals.yml
root@controller:~# grep -v ^# /etc/kolla/globals.yml | grep -v ^$
---
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
kolla_internal_vip_address: "192.168.100.11" # controller node IP
enable_haproxy: "no"
docker_registry: "192.168.100.10:5000"
network_interface: "ens3"
neutron_external_interface: "ens7"
designate_backend: "bind9"
designate_ns_record: "sample.openstack.org"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

edit multimode file.
root@controller:~# grep -v ^# multinode | grep -v ^$
[control]
controller
[network]
controller
[compute]
node01
node02
[monitoring]
controller
[storage]
controller
[baremetal:children]
control
network
compute
storage
monitoring
[chrony-server:children]
control
[chrony:children]
control
network
compute
storage
monitoring
[collectd:children]
compute
[grafana:children]
monitoring
[etcd:children]
control
[influxdb:children]
monitoring
[karbor:children]
control
[kibana:children]
control
[telegraf:children]
compute
control
monitoring
network
storage
[elasticsearch:children]
control
[haproxy:children]
network
[mariadb:children]
control
[rabbitmq:children]
control
[mongodb:children]
control
[keystone:children]
control
[glance:children]
control
[nova:children]
control
[neutron:children]
network
[cinder:children]
control
[cloudkitty:children]
control
[freezer:children]
control
[memcached:children]
control
[horizon:children]
control
[swift:children]
control
[barbican:children]
control
[heat:children]
control
[murano:children]
control
[solum:children]
control
[ironic:children]
control
[ceph:children]
control
[magnum:children]
control
[sahara:children]
control
[mistral:children]
control
[manila:children]
control
[ceilometer:children]
control
[aodh:children]
control
[congress:children]
control
[panko:children]
control
[gnocchi:children]
control
[tacker:children]
control
[trove:children]
control
[tempest:children]
control
[senlin:children]
control
[vmtp:children]
control
[watcher:children]
control
[rally:children]
control
[searchlight:children]
control
[octavia:children]
control
[designate:children]
control
[placement:children]
control
[glance-api:children]
glance
[glance-registry:children]
glance
[nova-api:children]
nova
[nova-conductor:children]
nova
[nova-consoleauth:children]
nova
[nova-novncproxy:children]
nova
[nova-scheduler:children]
nova
[nova-spicehtml5proxy:children]
nova
[nova-compute-ironic:children]
nova
[nova-serialproxy:children]
nova
[neutron-server:children]
control
[neutron-dhcp-agent:children]
neutron
[neutron-l3-agent:children]
neutron
[neutron-lbaas-agent:children]
neutron
[neutron-metadata-agent:children]
neutron
[neutron-vpnaas-agent:children]
neutron
[ceph-mon:children]
ceph
[ceph-rgw:children]
ceph
[ceph-osd:children]
storage
[cinder-api:children]
cinder
[cinder-backup:children]
storage
[cinder-scheduler:children]
cinder
[cinder-volume:children]
storage
[cloudkitty-api:children]
cloudkitty
[cloudkitty-processor:children]
cloudkitty
[freezer-api:children]
freezer
[iscsid:children]
compute
storage
ironic-conductor
[tgtd:children]
storage
[karbor-api:children]
karbor
[karbor-protection:children]
karbor
[karbor-operationengine:children]
karbor
[manila-api:children]
manila
[manila-scheduler:children]
manila
[manila-share:children]
network
[manila-data:children]
manila
[swift-proxy-server:children]
swift
[swift-account-server:children]
storage
[swift-container-server:children]
storage
[swift-object-server:children]
storage
[barbican-api:children]
barbican
[barbican-keystone-listener:children]
barbican
[barbican-worker:children]
barbican
[heat-api:children]
heat
[heat-api-cfn:children]
heat
[heat-engine:children]
heat
[murano-api:children]
murano
[murano-engine:children]
murano
[ironic-api:children]
ironic
[ironic-conductor:children]
ironic
[ironic-inspector:children]
ironic
[ironic-pxe:children]
ironic
[magnum-api:children]
magnum
[magnum-conductor:children]
magnum
[sahara-api:children]
sahara
[sahara-engine:children]
sahara
[solum-api:children]
solum
[solum-worker:children]
solum
[solum-deployer:children]
solum
[solum-conductor:children]
solum
[mistral-api:children]
mistral
[mistral-executor:children]
mistral
[mistral-engine:children]
mistral
[ceilometer-api:children]
ceilometer
[ceilometer-central:children]
ceilometer
[ceilometer-notification:children]
ceilometer
[ceilometer-collector:children]
ceilometer
[ceilometer-compute:children]
compute
[aodh-api:children]
aodh
[aodh-evaluator:children]
aodh
[aodh-listener:children]
aodh
[aodh-notifier:children]
aodh
[congress-api:children]
congress
[congress-datasource:children]
congress
[congress-policy-engine:children]
congress
[panko-api:children]
panko
[gnocchi-api:children]
gnocchi
[gnocchi-statsd:children]
gnocchi
[gnocchi-metricd:children]
gnocchi
[trove-api:children]
trove
[trove-conductor:children]
trove
[trove-taskmanager:children]
trove
[multipathd:children]
compute
[watcher-api:children]
watcher
[watcher-engine:children]
watcher
[watcher-applier:children]
watcher
[senlin-api:children]
senlin
[senlin-engine:children]
senlin
[searchlight-api:children]
searchlight
[searchlight-listener:children]
searchlight
[octavia-api:children]
octavia
[octavia-health-manager:children]
octavia
[octavia-housekeeping:children]
octavia
[octavia-worker:children]
octavia
[designate-api:children]
designate
[designate-central:children]
designate
[designate-mdns:children]
designate
[designate-worker:children]
designate
[designate-sink:children]
designate
[designate-backend-bind9:children]
designate
[placement-api:children]
placement
root@controller:~#

Generate passwords ( /etc/kolla/passwords.yml )
root@controller:~# kolla-genpwd

build images on the controller node and push images to the docker private registry.
root@controller:~# kolla-build --base ubuntu --type source --registry 192.168.100.10:5000 --push

Deploy OpenStack.
root@controller:~# kolla-ansible deploy -i ./multimode
TASK [Gathering Facts] ****************************************************************************************************
ok: [controller]

TASK [octavia : include] **************************************************************************************************
skipping: [controller]

PLAY RECAP ****************************************************************************************************************
controller                 : ok=225  changed=48   unreachable=0    failed=0
node01                     : ok=60   changed=23   unreachable=0    failed=0
node02                     : ok=60   changed=23   unreachable=0    failed=0

on controller node.
root@controller:~# docker ps
CONTAINER ID        IMAGE                                                                     COMMAND             CREATED             STATUS              PORTS               NAMES
7253548dbeef        192.168.100.10:5000/kolla/ubuntu-source-horizon:4.0.2                     "kolla_start"       3 minutes ago       Up 3 minutes                            horizon
ddf85fd52511        192.168.100.10:5000/kolla/ubuntu-source-heat-engine:4.0.2                 "kolla_start"       3 minutes ago       Up 3 minutes                            heat_engine
95ae0c6a5e10        192.168.100.10:5000/kolla/ubuntu-source-heat-api-cfn:4.0.2                "kolla_start"       3 minutes ago       Up 3 minutes                            heat_api_cfn
c404c7b0dd54        192.168.100.10:5000/kolla/ubuntu-source-heat-api:4.0.2                    "kolla_start"       3 minutes ago       Up 3 minutes                            heat_api
81e91bb064b3        192.168.100.10:5000/kolla/ubuntu-source-neutron-metadata-agent:4.0.2      "kolla_start"       4 minutes ago       Up 4 minutes                            neutron_metadata_agent
c6ad418b90b0        192.168.100.10:5000/kolla/ubuntu-source-neutron-l3-agent:4.0.2            "kolla_start"       4 minutes ago       Up 4 minutes                            neutron_l3_agent
42db0499ea32        192.168.100.10:5000/kolla/ubuntu-source-neutron-dhcp-agent:4.0.2          "koll

on node01
root@node01:~# docker ps
CONTAINER ID        IMAGE                                                                     COMMAND             CREATED             STATUS              PORTS               NAMES
27629bfbf3c1        192.168.100.10:5000/kolla/ubuntu-source-neutron-openvswitch-agent:4.0.2   "kolla_start"       4 minutes ago       Up 4 minutes                            neutron_openvswitch_agent
aeaa732e4a72        192.168.100.10:5000/kolla/ubuntu-source-openvswitch-vswitchd:4.0.2        "kolla_start"       5 minutes ago       Up 4 minutes                            openvswitch_vswitchd
17bbef6a2838        192.168.100.10:5000/kolla/ubuntu-source-openvswitch-db-server:4.0.2       "kolla_start"       5 minutes ago       Up 5 minutes                            openvswitch_db
5e59bf46dfd9        192.168.100.10:5000/kolla/ubuntu-source-nova-compute:4.0.2                "kolla_start"       10 minutes ago      Up 10 minutes                           nova_compute

on node02
root@node02:~# docker ps
CONTAINER ID        IMAGE                                                                     COMMAND             CREATED             STATUS              PORTS               NAMES
e8128ddf26f1        192.168.100.10:5000/kolla/ubuntu-source-neutron-openvswitch-agent:4.0.2   "kolla_start"       5 minutes ago       Up 5 minutes                            neutron_openvswitch_agent
108c2dcfcf63        192.168.100.10:5000/kolla/ubuntu-source-openvswitch-vswitchd:4.0.2        "kolla_start"       5 minutes ago       Up 5 minutes                            openvswitch_vswitchd
09b7ceabd1e6        192.168.100.10:5000/kolla/ubuntu-source-openvswitch-db-server:4.0.2       "kolla_start"       5 minutes ago       Up 5 minutes                            openvswitch_db
f6f534c61b22        192.168.100.10:5000/kolla/ubuntu-source-nova-compute:4.0.2                "kolla_start"       10 minutes ago      Up 10 minutes                           nova_compute


root@controller:~# kolla-ansible post-deploy

root@controller:~# ls /etc/kolla/admin-openrc.sh
/etc/kolla/admin-openrc.sh

root@controller:~# pip install python-openstackclient

root@controller:~# source /etc/kolla/admin-openrc.sh
root@controller:~# openstack service list
+----------------------------------+-------------+----------------+
| ID                               | Name        | Type           |
+----------------------------------+-------------+----------------+
| 0b727b6a11fc456495814b4381b38021 | nova_legacy | compute_legacy |
| 222bc4a9269d44adad625574acbeb94d | heat-cfn    | cloudformation |
| 2714d8f44c8d4391b4f2329ff1cd0e16 | heat        | orchestration  |
| 66c1c2f6a2cd42c9a67647f61defc8d6 | neutron     | network        |
| 67b757a4b3934a3cb7e9ef1ab65d8356 | glance      | image          |
| 696e48b1c7294bd28fa0017d69e26b88 | nova        | compute        |
| c4d8e948d9c14c9f8541d4f77bc361f9 | placement   | placement      |
| ef384109b76b4932bab8a43af38d4d40 | keystone    | identity       |
+----------------------------------+-------------+----------------+
root@controller:~#

add an image, create a network with a script.
root@controller:~# cd /usr/local/share/kolla-ansible/
root@controller:/usr/local/share/kolla-ansible# cp init-runonce init-runonce.orig

root@controller:/usr/local/share/kolla-ansible# diff init-runonce init-runonce.orig
11,13c11,13
< EXT_NET_CIDR='192.168.100.0/24'
< EXT_NET_RANGE='start=192.168.100.120,end=192.168.100.150'
< EXT_NET_GATEWAY='192.168.100.1'
---
> EXT_NET_CIDR='10.0.2.0/24'
> EXT_NET_RANGE='start=10.0.2.150,end=10.0.2.199'
> EXT_NET_GATEWAY='10.0.2.1'

run the script.
root@controller:/usr/local/share/kolla-ansible# source /etc/kolla/admin-openrc.sh
root@controller:/usr/local/share/kolla-ansible# ./init-runonce

Done.

To deploy a demo instance, run:

openstack server create \
   --image cirros \
   --flavor m1.tiny \
   --key-name mykey \
   --nic net-id=abf504e6-761d-4abc-b0a9-fd884c7122ca \
   demo1

access to the dashboard. ( controller’s IP )
you can find a user name and credentials at /etc/kolla/admin-openrc.sh.


on the controller node.
bring up br-ex.
root@controller:~# ip a s br-ex
728: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
   link/ether 3a:10:1d:8a:44:48 brd ff:ff:ff:ff:ff:ff
root@controller:~# ip link set br-ex up
root@controller:~# ip a s br-ex
728: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
   link/ether 3a:10:1d:8a:44:48 brd ff:ff:ff:ff:ff:ff
   inet6 fe80::3810:1dff:fe8a:4448/64 scope link
      valid_lft forever preferred_lft forever
root@controller:~#

launch two instances.


on the controller.
root@controller:~# source /etc/kolla/admin-openrc.sh

root@controller:~# openstack server list
+--------------------------------------+------------+--------+--------------------+--------+---------+
| ID                                   | Name       | Status | Networks           | Image  | Flavor  |
+--------------------------------------+------------+--------+--------------------+--------+---------+
| 0d3228af-722d-4728-a86e-a826f3f3cb82 | instance02 | ACTIVE | demo-net=10.0.0.6  | cirros | m1.tiny |
| 0dad6a28-29f8-411f-b0af-853cc67849d7 | instance01 | ACTIVE | demo-net=10.0.0.12 | cirros | m1.tiny |
+--------------------------------------+------------+--------+--------------------+--------+---------+
root@controller:~#

on the node01
root@node01:~# docker ps | grep libvirt
64deaef784ea        192.168.100.10:5000/kolla/ubuntu-source-nova-libvirt:4.0.2                "kolla_start"       18 hours ago        Up 18 hours                             nova_libvirt
root@node01:~#
root@node01:~# docker exec -t -i -u root 64deaef784ea virsh list
Id    Name                           State
----------------------------------------------------
1     instance-00000002              running

root@node01:~#

on the node02
root@node02:~# docker ps | grep libvirt
e845a1b8ea93        192.168.100.10:5000/kolla/ubuntu-source-nova-libvirt:4.0.2                "kolla_start"       18 hours ago        Up 18 hours                             nova_libvirt
root@node02:~# docker exec -t -i -u root e845a1b8ea93 virsh list
Id    Name                           State
----------------------------------------------------
1     instance-00000001              running

root@node02:~#

access to the neutron l3 agent.
on the controller
root@controller:~# docker ps | grep l3
c6ad418b90b0        192.168.100.10:5000/kolla/ubuntu-source-neutron-l3-agent:4.0.2            "kolla_start"       21 hours ago        Up 21 hours                             neutron_l3_agent

root@controller:~# docker exec -i -t -u root c6ad418b90b0 bash
(neutron-l3-agent)[root@controller /]# ip netns list
qrouter-804b79fc-426c-425b-8c71-6ba99db33b11
qdhcp-abf504e6-761d-4abc-b0a9-fd884c7122ca
(neutron-l3-agent)[root@controller /]#

root@controller:~# docker exec -i -t -u root c6ad418b90b0 bash
(neutron-l3-agent)[root@controller /]# apt install iputils-ping

(neutron-l3-agent)[root@controller /]# ip netns exec qrouter-804b79fc-426c-425b-8c71-6ba99db33b11 ping 10.0.0.6
64 bytes from 10.0.0.6: icmp_seq=1 ttl=64 time=2.02 ms
64 bytes from 10.0.0.6: icmp_seq=2 ttl=64 time=0.700 ms

add floating IPs

associate those floating IPs with two instances.

send pings to floating IPs.
root@controller:~# ping 192.168.100.129 -c 1
PING 192.168.100.129 (192.168.100.129) 56(84) bytes of data.
64 bytes from 192.168.100.129: icmp_seq=1 ttl=63 time=2.32 ms

--- 192.168.100.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.320/2.320/2.320/0.000 ms
root@controller:~#
root@controller:~# ping 192.168.100.125 -c 1
PING 192.168.100.125 (192.168.100.125) 56(84) bytes of data.
64 bytes from 192.168.100.125: icmp_seq=1 ttl=63 time=2.24 ms

--- 192.168.100.125 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.246/2.246/2.246/0.000 ms
root@controller:~#

SSH access via floating IP.
root@controller:~# ssh 192.168.100.125 -l cirros
$ hostname
instance01
$ ping 10.0.0.6 -c 3
PING 10.0.0.6 (10.0.0.6): 56 data bytes
64 bytes from 10.0.0.6: seq=0 ttl=64 time=1.728 ms
64 bytes from 10.0.0.6: seq=1 ttl=64 time=1.124 ms
64 bytes from 10.0.0.6: seq=2 ttl=64 time=0.964 ms

--- 10.0.0.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.964/1.272/1.728 ms
$
$ exit
Connection to 192.168.100.125 closed.
root@controller:~#

To destroy an environment, enter:
root@controller:~# kolla-ansible destroy --yes-i-really-really-mean-it

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.