Skip to main content
Blog

Managing multiple gears with Open Shift

By 4 juni 2015juli 4th, 2017No Comments

In the previous blog post we learned how we could scale an application horizontally. If you run your application on multiple gears, life becomes a bit more complicated. If you are investigating a production problem, the first thing in general to look at is the applications log file. Since there is now more than one instance, there are multiple log files as well, each on different machines. In this blog post I’ll show some tricks which can ease the pain a bit.

If you have a scaled web application with a database, your setup will look something like this:

 

scale2

Accessing log files

To find out what the names are of the gears, you can use rhc app show –gears:

$ rhc app show --gears -a mywebsite
ID                       State   Cartridges                                          Size  Region        Zone           SSH URL
------------------------ ------- --------------------------------------------------- ----- ------------- -------------- ------------------------------------------------------------------------
555c4d904382ece4d4000062 started jbossas-7 jbossas-7 jbossas-7 jbossas-7 haproxy-1.4 small aws-us-east-1 aws-us-east-1c 555c4d904382ece4d4000062@mywebsite-first8.rhcloud.com
555ddecc5973ca027b0000a8 started jbossas-7 jbossas-7 jbossas-7 jbossas-7             small aws-us-east-1 aws-us-east-1e 555ddecc5973ca027b0000a8@555ddecc5973ca027b0000a8-first8.rhcloud.com
555c501fe0b8cd41af00000d started postgresql-9.2                                      small aws-us-east-1 aws-us-east-1b 555c501fe0b8cd41af00000d@555c501fe0b8cd41af00000d-first8.rhcloud.com

Using this command you can see the gears ID’s, if they are running and what cartridges are running on them (in this case some JBoss AS 7 instances with a PostgreSQL database). Also note that you can access them directly via ssh using the ssh url’s.

You can also inspect the log files on a specific gear directly (without having to ssh into them first) using rhc tail:

$ rhc tail -g 555ddecc5973ca027b0000a8 -a mywebsite
==> app-root/logs/haproxy.log <==
[WARNING] 141/061212 (109007) : Server express/local-gear is DOWN, reason: Layer7 wrong status, code: 500, info: "HTTP status check returned code 500", check duration: 94ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 141/061224 (109007) : Server express/local-gear is UP, reason: Layer7 check passed, code: 200, info: "HTTP status check returned code 200", check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] 141/085052 (109007) : Server express/local-gear is DOWN for maintenance.
[WARNING] 141/085223 (109007) : Server express/local-gear is UP (leaving maintenance).
[WARNING] 141/085223 (109007) : Server express/gear-555ddecc5973ca027b0000a8-first8 is DOWN for maintenance.
[WARNING] 141/085323 (109007) : Server express/gear-555ddecc5973ca027b0000a8-first8 is UP (leaving maintenance).

==> app-root/logs/haproxy_ctld.log <==
I, [2015-05-22T04:56:39.917722 #260563]  INFO -- : Starting haproxy_ctld

Accessing specific gears

As described above, you can always access specific gears via ssh directly (or via your main gear and ssh from there).

In some cases it might be necessary to access the running application server on a specific gear using for example a browser. The problem is that HAProxy will load balance your request to a random gear that is available so you might end up on a different one that you want. In the example above, we could access the second gear (the one ending with …0a8) directly by visiting
http://555ddecc5973ca027b0000a8-first8.rhcloud.com. If we try that on the main gear (the one starting with …062) the request will be server by HAProxy and can thus end up anywhere.

A way around this is to create an ssh tunnel from your workstation towards the specific application ports on your gears. The Red Hat command line tools provide a convenient way of doing that:

$ rhc port-forward -a mywebsite
Checking available ports ... done
Forwarding ports ...
Address already in use - bind(2) while forwarding port 8080. Trying local port 8081
Address already in use - bind(2) while forwarding port 8080. Trying local port 8081
Address already in use - bind(2) while forwarding port 8081. Trying local port 8082

To connect to a service running on OpenShift, use the Local address
Service    Local                OpenShift
---------- --------------- ---- -----------------------------------------------------
haproxy    127.0.0.1:8081   =>  127.11.3.130:8080
haproxy    127.0.0.1:8082   =>  127.11.3.131:8080
java       127.0.0.1:3528   =>  127.11.3.129:3528
java       127.0.0.1:4448   =>  127.11.3.129:4447
java       127.0.0.1:5445   =>  127.11.3.129:5445
java       127.0.0.1:5455   =>  127.11.3.129:5455
java       127.0.0.1:8083   =>  127.11.3.129:8080
java       127.0.0.1:9991   =>  127.11.3.129:9990
java       127.0.0.1:10000  =>  127.11.3.129:9999
postgresql 127.0.0.1:55556  =>  555c501fe0b8cd41af00000d-first8.rhcloud.com:55556

Press CTRL-C to terminate port forwarding

Note that this creates port forwards for all processes on the main gear. The Java process on port 8080 is the JBoss web server which is forwarded from your desktop and is now reachable under <i>127.0.0.1:8083</i>. These forwards will exists for as long as you keep the rhc port-forward command running. Note that e.g. the PostgreSQL port on the third gear is also exposed now so you could use this to directly connect a sql client running on your desktop to the PostgreSQL running on Open Shift by using the address 127.0.0.1:55556. There are also two HAProxy instances forwarded, one for each JBoss instance on the two gears, as well as a bunch of other internal JBoss modules.

You can also setup tunnels per gear:

sune:~ arjanl$ rhc port-forward -a mywebsite -g 555ddecc5973ca027b0000a8
Checking available ports ... done
Forwarding ports ...
Address already in use - bind(2) while forwarding port 3528. Trying local port 3529
Address already in use - bind(2) while forwarding port 4447. Trying local port 4448
Address already in use - bind(2) while forwarding port 4448. Trying local port 4449
Address already in use - bind(2) while forwarding port 5445. Trying local port 5446
Address already in use - bind(2) while forwarding port 5455. Trying local port 5456
Address already in use - bind(2) while forwarding port 8080. Trying local port 8081
Address already in use - bind(2) while forwarding port 8081. Trying local port 8082
Address already in use - bind(2) while forwarding port 8082. Trying local port 8083
Address already in use - bind(2) while forwarding port 8083. Trying local port 8084
Address already in use - bind(2) while forwarding port 9990. Trying local port 9991
Address already in use - bind(2) while forwarding port 9991. Trying local port 9992
Address already in use - bind(2) while forwarding port 9999. Trying local port 10000
Address already in use - bind(2) while forwarding port 10000. Trying local port 10001

To connect to a service running on OpenShift, use the Local address

Service Local                OpenShift
------- --------------- ---- ----------------
java    127.0.0.1:3529   =>  127.13.96.1:3528
java    127.0.0.1:4449   =>  127.13.96.1:4447
java    127.0.0.1:5446   =>  127.13.96.1:5445
java    127.0.0.1:5456   =>  127.13.96.1:5455
java    127.0.0.1:8084   =>  127.13.96.1:8080
java    127.0.0.1:9992   =>  127.13.96.1:9990
java    127.0.0.1:10001  =>  127.13.96.1:9999

Press CTRL-C to terminate port forwarding

Now you can directly access both application servers explicitly, in this example via 127.0.0.1:8083 for the main gear and 127.0.0.1:8084 for the second gear.

Using these commands, it is a lot easier to manage multiple gears.

Read more