In the previous blog post we learned how we could scale an application horizontally. If you run your application on multiple gears, life becomes a bit more complicated. If you are investigating a production problem, the first thing in general to look at is the applications log file. Since there is now more than one instance, there are multiple log files as well, each on different machines. In this blog post I’ll show some tricks which can ease the pain a bit.
If you have a scaled web application with a database, your setup will look something like this:
Accessing log files
To find out what the names are of the gears, you can use rhc app show –gears:
$ rhc app show --gears -a mywebsite ID State Cartridges Size Region Zone SSH URL ------------------------ ------- --------------------------------------------------- ----- ------------- -------------- ------------------------------------------------------------------------ 555c4d904382ece4d4000062 started jbossas-7 jbossas-7 jbossas-7 jbossas-7 haproxy-1.4 small aws-us-east-1 aws-us-east-1c 555c4d904382ece4d4000062@mywebsite-first8.rhcloud.com 555ddecc5973ca027b0000a8 started jbossas-7 jbossas-7 jbossas-7 jbossas-7 small aws-us-east-1 aws-us-east-1e 555ddecc5973ca027b0000a8@555ddecc5973ca027b0000a8-first8.rhcloud.com 555c501fe0b8cd41af00000d started postgresql-9.2 small aws-us-east-1 aws-us-east-1b 555c501fe0b8cd41af00000d@555c501fe0b8cd41af00000d-first8.rhcloud.com
Using this command you can see the gears ID’s, if they are running and what cartridges are running on them (in this case some JBoss AS 7 instances with a PostgreSQL database). Also note that you can access them directly via ssh using the ssh url’s.
You can also inspect the log files on a specific gear directly (without having to ssh into them first) using rhc tail:
$ rhc tail -g 555ddecc5973ca027b0000a8 -a mywebsite ==> app-root/logs/haproxy.log <== [WARNING] 141/061212 (109007) : Server express/local-gear is DOWN, reason: Layer7 wrong status, code: 500, info: "HTTP status check returned code 500", check duration: 94ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] 141/061224 (109007) : Server express/local-gear is UP, reason: Layer7 check passed, code: 200, info: "HTTP status check returned code 200", check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. [WARNING] 141/085052 (109007) : Server express/local-gear is DOWN for maintenance. [WARNING] 141/085223 (109007) : Server express/local-gear is UP (leaving maintenance). [WARNING] 141/085223 (109007) : Server express/gear-555ddecc5973ca027b0000a8-first8 is DOWN for maintenance. [WARNING] 141/085323 (109007) : Server express/gear-555ddecc5973ca027b0000a8-first8 is UP (leaving maintenance). ==> app-root/logs/haproxy_ctld.log <== I, [2015-05-22T04:56:39.917722 #260563] INFO -- : Starting haproxy_ctld
Accessing specific gears
As described above, you can always access specific gears via ssh directly (or via your main gear and ssh from there).
In some cases it might be necessary to access the running application server on a specific gear using for example a browser. The problem is that HAProxy will load balance your request to a random gear that is available so you might end up on a different one that you want. In the example above, we could access the second gear (the one ending with …0a8) directly by visiting
http://555ddecc5973ca027b0000a8-first8.rhcloud.com. If we try that on the main gear (the one starting with …062) the request will be server by HAProxy and can thus end up anywhere.
A way around this is to create an ssh tunnel from your workstation towards the specific application ports on your gears. The Red Hat command line tools provide a convenient way of doing that:
$ rhc port-forward -a mywebsite Checking available ports ... done Forwarding ports ... Address already in use - bind(2) while forwarding port 8080. Trying local port 8081 Address already in use - bind(2) while forwarding port 8080. Trying local port 8081 Address already in use - bind(2) while forwarding port 8081. Trying local port 8082 To connect to a service running on OpenShift, use the Local address Service Local OpenShift ---------- --------------- ---- ----------------------------------------------------- haproxy 127.0.0.1:8081 => 127.11.3.130:8080 haproxy 127.0.0.1:8082 => 127.11.3.131:8080 java 127.0.0.1:3528 => 127.11.3.129:3528 java 127.0.0.1:4448 => 127.11.3.129:4447 java 127.0.0.1:5445 => 127.11.3.129:5445 java 127.0.0.1:5455 => 127.11.3.129:5455 java 127.0.0.1:8083 => 127.11.3.129:8080 java 127.0.0.1:9991 => 127.11.3.129:9990 java 127.0.0.1:10000 => 127.11.3.129:9999 postgresql 127.0.0.1:55556 => 555c501fe0b8cd41af00000d-first8.rhcloud.com:55556 Press CTRL-C to terminate port forwarding
Note that this creates port forwards for all processes on the main gear. The Java process on port 8080 is the JBoss web server which is forwarded from your desktop and is now reachable under <i>127.0.0.1:8083</i>. These forwards will exists for as long as you keep the rhc port-forward command running. Note that e.g. the PostgreSQL port on the third gear is also exposed now so you could use this to directly connect a sql client running on your desktop to the PostgreSQL running on Open Shift by using the address 127.0.0.1:55556. There are also two HAProxy instances forwarded, one for each JBoss instance on the two gears, as well as a bunch of other internal JBoss modules.
You can also setup tunnels per gear:
sune:~ arjanl$ rhc port-forward -a mywebsite -g 555ddecc5973ca027b0000a8 Checking available ports ... done Forwarding ports ... Address already in use - bind(2) while forwarding port 3528. Trying local port 3529 Address already in use - bind(2) while forwarding port 4447. Trying local port 4448 Address already in use - bind(2) while forwarding port 4448. Trying local port 4449 Address already in use - bind(2) while forwarding port 5445. Trying local port 5446 Address already in use - bind(2) while forwarding port 5455. Trying local port 5456 Address already in use - bind(2) while forwarding port 8080. Trying local port 8081 Address already in use - bind(2) while forwarding port 8081. Trying local port 8082 Address already in use - bind(2) while forwarding port 8082. Trying local port 8083 Address already in use - bind(2) while forwarding port 8083. Trying local port 8084 Address already in use - bind(2) while forwarding port 9990. Trying local port 9991 Address already in use - bind(2) while forwarding port 9991. Trying local port 9992 Address already in use - bind(2) while forwarding port 9999. Trying local port 10000 Address already in use - bind(2) while forwarding port 10000. Trying local port 10001 To connect to a service running on OpenShift, use the Local address Service Local OpenShift ------- --------------- ---- ---------------- java 127.0.0.1:3529 => 127.13.96.1:3528 java 127.0.0.1:4449 => 127.13.96.1:4447 java 127.0.0.1:5446 => 127.13.96.1:5445 java 127.0.0.1:5456 => 127.13.96.1:5455 java 127.0.0.1:8084 => 127.13.96.1:8080 java 127.0.0.1:9992 => 127.13.96.1:9990 java 127.0.0.1:10001 => 127.13.96.1:9999 Press CTRL-C to terminate port forwarding
Now you can directly access both application servers explicitly, in this example via 127.0.0.1:8083 for the main gear and 127.0.0.1:8084 for the second gear.
Using these commands, it is a lot easier to manage multiple gears.
