Hardware Regions
Click on the Regions option in your admin panel’s Cluster menu section to see the hardware structure of the platform:
- Regions (or hardware regions) - independent hardware sets from different data centers; each region can contain multiple host groups
- Host Groups (or environment regions) - a separate set of servers (hosts) within the confines of a particular region with its own options, efficiency, and rules for resources charging
Here, all the crucial information on Regions is displayed through the following columns:
- Name of a hardware region or comprised host group(s)
- Domain assigned to the region
- SSL certificates configuration for the hardware region
- Subnet provided for the region
- Migration shows if users should be able to migrate environments from/to the current hardware region
- Status of a region/host group (could be either ACTIVE or under MAINTENANCE)
- Comment with some optional information on a region or host group
- Docker Host address of the hardware region
Tip: If you want to benefit from providing multiple regions, you should read the appropriate documentation before applying any changes:
Use the tools panel above the regions list to perform the following operations:
Also, you can select a particular region to view its detailed information and manage domain names.
Add New Region
Follow the next steps to add a new hardware region to your PaaS cluster:
Note: Before adding a new region, consider the following prerequisites:
- hosts must be configured according to hardware requirements
- at least two internal and two external IPs must be reserved for shared load balancers (resolvers)
- new region domain delegation must be done to the IPs from the previous point and according to DNS Zones Delegation
- firewall should be checked and, if necessary, set up
- In order to set up the internal and external networks in the new region, you need to fulfill the following two conditions:
- Internal and external interfaces on the server should not interconnect. You can connect these interfaces to different physical switches in the data center or separate them on the data center network level (Vlan).
- Every host’s internal and external network interface must be configured in the same segment of the data center network (Vlan or Vrack). For example, all internal interfaces of the region - vlan1, all external - vlan2 (i.e. no interconnection).
1. Click the Add Region button at the top pane of the Regions section:
Within the opened Add Region frame, you need to fulfill the required details.
2. Within the first Region Setting section, specify the following information:
Unique Name - unique identifier for the region (cannot be changed later)
Display Name - changeable region alias, which is displayed in JCA (10 characters max)
Domain - hostname assigned to a new region
Note: The appropriate domain name should be purchased beforehand using any preferred domain registrar.Status - the initial state should be set as MAINTENANCE to avoid false monitoring alerts during region addition
Subnet - a dedicated internal subnet for the user nodes and traffic routing between different hardware regions
Start and End IP - range of the IP addresses for containers created in this region (cannot exceed the specified subnet)
Comment - short information on the current hardware region displayed in the admin panel (optional)
Allow migration from/to regions - tick the checkbox to allow environments migration from/to this region by end-users
Note: This parameter controls the permission for migration across different hardware regions; herewith, transferring between host groups of the same region cannot be disabled.
3. In the Name Servers section, you need to state a pair (or several pairs) of Public IPv4 and Internal IPv4. These addresses will be used by shared load balancers as a region entrance point and, at the same time, its internal and external DNS server.
4. The last Docker Host Settings section configures a separate Docker Engine module for this particular hardware region:
- Host - domain or IP of your Docker Host
- SSH and TCP Port - ports for connections via the appropriate protocols
- Login and Password - access credentials for the Docker Host
Once all the settings are specified, confirm the creation by clicking the Add button.
Add New Host Group
To add a new host group, follow the instructions below.
1. Click the Add Host Group button at the top of the Regions panel.
2. Within the opened Add Host Group dialog, fill in the given fields on the required Basic Data tab:
- Unique Name - unique identifier for the host group (cannot be changed later)
- Display Name - changeable host group name displayed in the admin panel and at the end-users' dashboard (10 characters max)
- Status - initial state of the host group, i.e. the one set after creation (ACTIVE or MAINTENANCE)
- Comment - short information on the current host group displayed in the admin panel (optional)
- Region - hardware region this host group should be assigned to (use the drop-down list to select an existing one or to jump to the Add Region dialog)
- Virtual Network Group (VNG) - a value used for the virtual network grouping (host groups with the same VNG will have the same configs)For example: You have two regions (for infrastructure containers and user applications), which are physically in the same data center. If you set “Virtual Network Group (VNG)” for all host groups of both regions to the same value, the virtual network configs will be created as if for the same region.
Optionally, provide additional information via the Advanced Settings tab:
- Datacenter
- Country – choose the country of the datacenter
- City – specify the city of the datacenter
- Geo Coordinates – set latitude and longitude coordinates of datacenter in the WGS84 decimal degrees format
- Vendor – specify the datacenter’s vendor name
- Dashboard Settings
- Icon 16x16 – provide URL or use the Browse button to select from the platform resources folder; if not provided, country flag (based on the selected above) is used by default
- Short Description – write a single-line description
- Description (markdown) - prepare a detailed description for the host group
Click Add to proceed.
3. Next, you need to set up internal routing between regions by following the steps described in the linked section. Skip this step if it is already configured.
4. Add a host to this newly created host group.
4.1. Check /etc/vz/vz.conf. If the VE_ROUTE_SRC_DEV parameter is commented or indicates an incorrect device, fix the issue and save the file. It should specify the interface name of the server’s internal network.
4.2. If your DOCKER_HOST is on the docker-engine host and you deploy vz7 host, add the next line to the /etc/yum.conf file:
|
|
4.3. Check routes from the new region to infra/user hosts in this and other regions. It could be set automatically via the bird daemon.
4.4. Start the host installation via JCA.
5. Configure shared load balancers (SLB).
5.1. Add a region network to the jelastic.net.subnetworks system settings in JCA.
5.2. Add SLB IPs (both external and internal) to the jelastic.isolation.infra.ips and jelastic.isolation.infra.ips.all system settings in JCA. If isolation is enabled on the platform, you need to disable and re-enable it to apply these new settings.
5.3. In order to create a shared load balancer for the new region, connect to a new host and create the config.ini file:
|
|
The ${PLATFORM_NETWORK} is the primary platform network of the main region.
5.4. Download the create_docker.sh script.
|
|
Edit it to specify the platform version in the DOCKER_VERSION=""; line.
For example, if deploying a region to PaaS 5.9-3, set it as follows: DOCKER_VERSION=“5.9-3”;
5.5. Run the script to create a new shared load balancer.
|
|
Add all regions' networks to this SLB via the /var/lib/jelastic/customizations/ipconfig.cfg file.
5.6. Update ZooKeeper environment variables (/.jelenv) by adding shared load balancer’s internal IP to OPT_JELASTIC_IPS and new network to JELASTIC_NETWORK. Restart the ZooKeeper service to apply changes:
|
|
5.7. Fix nameservers for SLB containers.
|
|
5.8. Check all infrastructure containers and manually add region network and routes.
|
|
5.9. Check the connection between the SLB and database containers via port 3306.
Tip: If there is no connection, you can temporarily add the following route to the containers:
|
|
Here, {db_network} is the database container network (e.g. 10.100.0.0/16), and {slb_ip} is SLB internal address (e.g. 10.103.1.2).
Run service discovery:
|
|
Check results in /vz/root/$new_resolver_CTID/var/log/discovery.log and, if everything is ok, disable discovery:
|
|
5.10. If adding a new shared load balancer(s) to the current region, you need to add the appropriate IPs to the virtual_common.conf file inside the HCore containers:
|
|
6. Provide Let’s Encrypt SSL certificates via JCA.
7. If needed, apply customizations and run J-runner tests for the new region.
8. Synchronize new SLBs in the patcher.
Under the patcher account, you need to run the “Sync infra components” and “Zabbix updater” JPS scripts from the Marketplace. It will automatically add new SLBs to the Zabbix server.
9. Finally, assign the host group to the appropriate user Groups via the Regions & Pricing tab.
Afterward, your host group will appear in the topology wizard of the Dev dashboard as a new environment region.
Internal Routing between Regions
To connect a new region with the current platform, you need to configure GRE + IPsec tunnels and set up internal routing with the BIRD daemon.
1. Set up GRE tunnels.
You should perform the following configurations between two infrastructure hosts and the first two user hosts in the region (for high availability).
List of the tunnels to be configured:
|
|
1.1. Create a configuration file with corresponding hostnames and IP addresses (on all the hosts participating in GRE tunneling):
|
|
Here:
- 1st column - list of valid hostnames assigned to the hosts that are participating in GRE tunneling
- 2nd column - index number of the host
- 3rd column - abbreviated location of the hosts (e.g. OVH = ovh, User = usr); don’t use more than three letters, only lowercase with no special symbols
- 4th column - inner GRE IP addresses (preferably from the 100.127.255.1-100.127.255.254 range)
- 5th column - external IPv4 addresses of the hosts (obtain with the “ip r g 1 | awk {‘print $NF’} | head -n 1” command)
1.2. Generate configs using the script and start the GRE link interfaces. Execute commands listed below on the required hosts:
- the first infra host
|
|
Start GRE tunnels.
|
|
Disable reverse path filtering for the host.
|
|
- the second infra host
|
|
Start GRE tunnels.
|
|
Disable reverse path filtering for the host.
|
|
- the first user host
|
|
Start GRE tunnels.
|
|
Disable reverse path filtering for the host.
|
|
- the second user host
|
|
Start GRE tunnels.
|
|
Disable reverse path filtering for the host.
|
|
The hosts at both ends should successfully ping each other (use the IP addresses from the 4th column) to ensure that all links are set up correctly.
2. Set up IPsec tunnels.
2.1. Install the required libreswan software.
|
|
2.2. Put the following config template into the /etc/ipsec.conf file on each host.
|
|
2.3. Generate IPsec key for all hosts participating in tunneling.
|
|
As a result of these commands, you’ll get a key. In this guide, we’ll refer to it as $(infranode key).
Repeat this step on the user hosts of the new region to get $(usernode key).
2.4. Create the following configs:
- /etc/ipsec.d/default.conf on user hosts
|
|
- /etc/ipsec.d/$(infranode hostname).conf on user hosts
|
|
- /etc/ipsec.d/default.conf on infra hosts
|
|
- /etc/ipsec.d/$(usernode hostname).conf on infra hosts
|
|
2.5. Restart IPsec and enable it on each host.
|
|
2.6. Create IPsec connections on:
- user host
|
|
- infra host
|
|
Note: Before proceeding to the next step, run the following command to ensure that IPsec is set:
|
|
You should see active tunnels in the output, for example:
|
|
2.7. Set up the firewall rules on the new hosts and add some firewall rules on the infra hosts.
|
|
Here:
- Internal_IP_address_host - internal IP address of the host
- br1 - a name of the internal network interface for local connections
- IP_link_infra1, IP_link_infra2, IP_link_user1, IP_link_user2 - external IP addresses of hosts that participate in tunneling
2.8. Add the following rules to other user hosts that are in the cluster but did not participate in configuring IPsec tunnels:
|
|
2.9. Check if IPsec tunnels are working. Execute the command below on any host and ensure that the packets go between the hosts.
|
|
If everything works fine, please save the firewall rules on each node.
|
|
3. Set up BIRD.
You can automatically set up internal routing with the BIRD daemon.
3.1. Install BIRD on the infra and user hosts:
|
|
3.2. Set up BIRD via the /etc/bird.conf configuration file.
|
|
Here:
- Internal_IP_address_host - the internal IP address of the host
- anypassword - generate any password for your BIRD connection
- br1 - internal network interface for local connections
3.3. If you do not use IPsec + GRE, remove the following code block from the BIRD configurations:
|
|
3.4. Restart the BIRD service and ensure that other hosts are present in the Full/PtP state. For example:
|
|
3.5. Manually test the connection from the infra to user hosts via the internal IP (in both directions).
3.6. Fill in the jelastic.net.subnetworks system setting in JCA and verify that all internal networks are present for all platform regions.
For example: 172.16.0.0/16;10.30.0.0/16.
That’s all. You can proceed with the host group configuration (the fourth step).
Edit Region/Host Group
You can adjust the existing regions and host groups by simply double-clicking on the required item or using the Edit button at the top of the Regions panel.
Within the corresponding region/host group Edit dialog, you can adjust everything on both tabs (same as for the addition) except the Unique Name value.
Apply changes with the Save button at the bottom-right corner of the frame.
SSL Certificates for Regions
Using the SSL column within the Regions section, you can configure SSL certificates of the primary domain (go to the dedicated section to manage all domains):
- Add Certificates – sets up SSL for the region
- Edit - allows switching between the Let’s Encrypt and custom SSL certificates
- Update - provides a new LE certificate for the hardware region (this option is hidden for custom SSL)
- Remove - detaches certificate from the region
1. While adding or editing your certificate, you can choose between two options:
- Use Let’s Encrypt - automatically fetch and apply certificates from the Let’s Encrypt free and open Certificate Authority
- Upload Custom Certificates - upload valid RSA-based Server Key, Intermediate Certificate (CA), and Domain Certificate files to automatically apply them. Self-signed certificates can be used as well, e.g. for testing purposes
Click Save to confirm changes.
2. If needed, you can configure the Let’s Encrypt certificates provisioning via the certain System Settings:
- jelastic.letsencrypt.renewal.days - displays an alert at JCA if any of the SSL certificates are valid for fewer days than a provided value (21, by default)
- qjob.ssl_checker.cron_schedule - checks the status of the Let’s Encrypt SSL certificates for hardware regions and automatically updates those, which are valid for fewer days than specified in the jelastic.letsencrypt.renewal.days setting; the default value is 0 0 15 * * ?, i.e. this job is run daily at 15:00
- hcore.platform.admin.username - sets platform admin email address, which, in case any issue occurs, receives notification from Let’s Encrypt
To update or remove a certificate, select the appropriate option from the list, and confirm the action via the pop-up window.
Remove Region/Host Group
No longer needed regions and host groups can be deleted with the help of the Remove button at the top tools panel.
Confirm your decision via the pop-up window.