..:: JOCHUS ONLINE! ::..

I recently installed iTerm2 on my Mac, and I must say: "WOW! What a difference!"

The most popular features for me are:

Split Panes

Divide a tab up into multiple panes, each one of which shows a different session. You can slice vertically and horizontally and create any number of panes in any imaginable arrangement.

Search

iTerm2 comes with a robust find-on-page feature. The UI stays out of the way. All matches are immediately highlighted. Even regular expression support is offered!

Mouseless copy

Use the Find feature to begin searching for text. Press tab to expand the selection to the right or shift-tab to expand the selection to the left. Option-enter pastes the current match.

Configurability

Map any key to any function. Assign separate functions to each option key--or even remap all the modifier keys. You can customize iTerm2's appearance to suit your needs: enable transparency, background blur, background images, and much more.

Unixyness

Coming from a Unix world? You'll feel at home with focus follows mouse, copy on select, middle button paste, and keyboard shortcuts to avoid mousing.

Growl Support

You can choose to receive Growl notifications of activity, bells, and more. Feel free to let a long job run in the background, secure in the knowledge that you'll know when it's done.


You can find more information on: http://www.iterm2.com/#/section/home

Well, I had the possibility to "taste" from both worlds. I started at Dolmen (which is now known as RealDolmen). I got a specialized training in all kinds of Java frameworks. After 3.5 years, I was getting really pro in some of the JBoss frameworks (Seam, Hibernate, Richfaces ...). Also, EJB and JSF are frameworks which have almost no secrets left for me.

After RealDolmen, I started at Pigeon Paradise. A company very close to home, which gave me the possibility to have more free time to learn more things on my own. I had the possibility to play with Drupal at PIPA. I was really looking forward to this new technology. After a 1.5 year, I got specialized in Drupal 6. I've created a lot of custom modules, together with 2 other Drupal developers. I do have a lot of experience in working in a team on Drupal projects.

So, as you can read, I had experience with both worlds. I would like to compare the 2 "worlds" in this blogpost.

Drupal

The power of a scripting language

The main advantage I see on Drupal is actually an advantage of PHP. It's a scripting language (Java needs to be compiled to a binary file, afterwards, you can run that binary file in a JVM). When you develop in PHP/Drupal, you don't need to compile your source code. And most of all, it's very, very easy to just do whatever you want to do.
For example:

$content = '';
$result = db_query("SELECT foo FROM bar");
while ($row = db_fetch_object($result)) {
    $content .= theme("list-item", $row->foo);
}
 
return theme("list", $content);

So what you can see here, is a wrap of a database query, some (simple) business logic and a call to a theming function. It's really possible to program amazingly fast. If you made a typo, you fix it, save the source file, push F5 in your browser and tadaaaaa ... the new result.

Design pattern: Model-View-Controller

In my opinion, Drupal has an implementation of the MVC pattern. The hook_menu() is actually the controller and delegating all the requests to the correct page callback. In the page callback, you can perform some business logic and you can ask the theming function to wrap your model in a pretty jacket.


But there are things I'm missing in Drupal, and ... which Java is having.

Java

Build/release management tool

This issue came up because Java has a buildtool like Maven or Ant. Drupal has Drush. Drush is nice, but it needs some "team" features to control Drupal projects. It's not simple to control the lifecycles of custom Drupal modules, or the core of Drupal. I noticed D.O. has some kind of releasemanagement, but I wasn't able to set it up for our company. You cannot easily create branches, tags, trunk, ... You need a whole custom setup to achieve this. I once gave a presentation about this for drupal.be. Please check some files here: http://www.jochus.be/site/files/. It's hard stuff, because the core has his lifecycle, but contrib and custom Drupal modules have them too. You must keep them seperated.

Integration with IDE

Eclipse (or IntelliJ, or whatever Java IDE) and Java just match... They match great. There are so many nice plugin to integrate (such as JBoss Tools for JBoss frameworks), which can make development so easy. It actually makes programming very fun! In Drupal, I was missing some great plugins. I also worked with Eclipse (and PHPDesigner), but the IDE isn't just integrated with your project. If you want to debug your PHP application, it takes a while to set up the server as client side. In Java (well, actually ... Jboss), this debugging is going very easy, especially with Eclipse.

GUI library framework

Well, I have been using Richfaces a lot. I must say: I really miss this framework in Drupal :-( ...

RichFaces is an open source Ajax-enabled component library for JavaServer Faces, hosted by JBoss. It allows easy integration of Ajax capabilities into enterprise application development. RichFaces is more than just a component library for JavaServer Faces. It adds:

• Skinability (easily change and update application look and feel)
• Component Development Kit (CDK) to assist in constructing JavaServer Faces components
• Dynamic Resource Framework
• Both page wide, and component based Ajax control components.

And of the things not mentioned above: the UI components WORK in different browsers. Drupal is still missing this GUI layer to easily create some pages. I'm not interested in writing custom HTML elements, nor in moving an input element 5 pixels to the left in Chrome, but 5 pixels to the right in IE ... No, I just want a < rich:tree > or < rich:datatable >. And this component should work in whatever browser you're trying to open it!

Object oriented programming

Drupal 6 isn't written in OO. I've noticed Drupal 7 got some improvements on this issue, but still ... there's a hugh lack of OO design. I miss this a lot. OO has following advantages according to me:

• simplicity: software objects model real world objects, so the complexity is reduced and the program structure is very clear
• modularity: each object forms a separate entity whose internal workings are decoupled from other parts of the system
• modifiability: it is easy to make minor changes in the data representation or the procedures in an OO program. Changes inside a class do not affect any other part of a program, since the only public interface that the external world has to a class is through the use of methods
• extensibility: adding new features or responding to changing operating environments can be solved by introducing a few new objects and modifying some existing ones
• maintainability: objects can be maintained separately, making locating and fixing problems easier
• re-usability: objects can be reused in different programs

3-tier architecture

For me, the greatest "pain-in-the-ass" in Drupal. I do think you need to separate database calls and service calls. In a certain project of my previous company, I noticed the same query turning up 3 times in 3 different files, but they were actually doing the same :-/ ... I do think you should create DAO's. Combine all functions which are related to a specific domain object and group all queries together into one single interface. Next, call the function on this interface from your service layer. And let your service layer call your view layer. The link service/view layer is kind of present in Drupal, but Java just beats Drupal by using JPA (Hibernate) / EJB / Seam / JSF and Richfaces.

Amount of stable frameworks

In Java, I think there are 2 famous frameworks to work with: Spring projects and Seam projects. And that's it. There's a lot of information about these frameworks on the internet. And some really large communities.
In Drupal - you are looking for a specific kind of framework, you can find a lot of "frameworks". They are called contrib modules. For a specific kind of functionality, you can have 4 or 5 contrib modules. But none of them actually does what it has to do. And the most painful thing: some contrib modules are just placed online, but there's no working community behind the product any longer. So if you have some issues/problems/questions, nobody can help you out.

Conclusion

For me, Drupal isn't ready to handle real business applications. It's a good product, but I would only recommend it for really simple projects. To summarize Drupal: get input data from the user, store it in the database and retrieve it afterwards. I sometimes miss the complexity of programming in Drupal. Like BPM, a service bus layer, SOA, ORM, ...
I also think I didn't receive much time/space to discover all those things as a Drupal developer (or to start implement them - if you don't have it, you can try to create it) in my previous company. I cannot blame them: they are not an IT company. Their core business is selling pigeons, not creating a Drupal product and all other kinds of tools to work with it. If I want that, I should start at Acquia :-). I do think Drupal can achieve that level (which is Java having now) after a lot of effort. But it's gonna need time and coordination. And a lot of patience ...

Anyways, after 1.5 year working real hard on Drupal, I had to give up. I was getting too much frustrated in the way of working. It might have been my previous company's way of working, I don't know. But I knew I was missing Java. And after 1.5 years, I switched back to Java. I'm now programming for almost 4 weeks back in Java, and believe me: I HAVE BEEN MISSING THIS A LOT!

A long time ago, I made a blogpost about a solution for Drupal and a MASTER-MASTER replication setup: http://www.jochus.be/site/2011-01-13/drupal/drupal-master-master-replica.... At that moment of writing, I was very happy with that solution. But after some months of experience, I must say, I cannot agree any longer with my own solution. Some people were commenting that blogpost, and I really wanted to post my point of view of a new possible architecture. I definitely don't want to spread around a bad architecture setup. The new environment holds some improvements (which I cover at the bottom of this blogpost). To be honest, I must say that I only implemented the memcache improvement of the new architecture. There was no time/budget in my previous company for the other improvements, but I strongly believe those extra improvements are the way to handle a MASTER-MASTER replication setup.

Why is the old architecture so bad?

In the matter of fact, we were having too much issues with the old architecture. We were losing time and the architecture was just unstable. I'll explain why:

• The caching mechanism was initially configured to use the MySQL databases. So all CRUD operations on caching tables were executed towards the MySQL databases. Each web instance was having his own caching mechanism - we didn’t replicate the cache tables (replicate-ignore-table setting in /etc/mysql/my.cnf). The tables were handling string keys as primary keys, so it would be impossible to avoid conflicts. As a small reminder: you can only set even or odd integer numbers per MySQL instance to autoincrement ID numbers. So we had to switch to a mechanism where both instances were having their own cache tables, to avoid duplicate SQL entries.
• However, Drupal is writing so many things to the cache tables, the MySQL BIN log files were really getting overloaded. As I can remember, we had every 7 or 8 minutes a new MySQL BIN log file of 100MB. And every time a Drupal node_save() was called, "all" cache was just blown away ... I still don’t believe this is a very efficient way of flushing the cache. I tried a conversation on D.O.: http://drupal.org/node/115319#comment-5243634, but as you can read, changes would only effect Drupal 8.
• There was no replication system to flush the remote cache on the other web instance. Therefore, we had to write a custom mechanism. This mechanism has been explained in my other blogpost (I’m not gonna repeat it again). I still doubt this mechanism was working efficient ... It was working yes, but still ... Very amateurish.
• We were having issues with duplicate SQL entries on the semaphore table. The reason why is again explained in the previous blogpost. Solution can be found over there either. But this was still introducing us some other problem. Now Drupal could rebuild the table menu_router at both instances at the same time. The lock to this rebuilding mechanism is created by an entry in the semaphore table. But this lock wasn't replicated as we didn't replice the the semaphore table. menu_router is using the path column as a primary key. path is varchar. So again, we received duplicate SQL entries on this one. The only way to solve this problem was to manually skip the blocking SQL queries (which you don’t like doing when it happens at 3 O’Clock in the night :-) ...). A temporary solution was to whenever we wanted to rebuild the menu_router table (e.g.: you are flushing the menu cache, and believe me, this can happen a lot, even when you don’t know it), shutting down the second web instance. Wait for all traffic to be gone. If there's no traffic on the second web instance, the second web instance will not be rebuilding the menu_router table. Luckily, the load balancer would switch all traffic for us. Sessions are replicated too, so don’t worry about that one either. Then we could flush the menu cache. And when menu_router has been rebuilt, we could bring the second web instance again up.
• We were copying content files (which are located in /sites/##YOUR_SITE_NAME##/files) from server 1 to server 2 (and vice versa) using rsync. As rsync couldn't know if we were deleting a file, or if we were adding a file, we didn't allow deletes. Our htdocs folder was growing really fast. The monthly full backup was taking a lot of gigabytes.

The new architecture

So, after all these problems, we came up (also thanks to the hosting provider of my previous company) to this new setup which seems to me a better solution:

Memcache

• We moved the caching mechanism to memcache. To know more about Drupal and memcache, read the documentation on: http://drupal.org/project/memcache. Memcache can store all caching data in memory. So it doesn't use the MySQL tables any longer. Memcache is just way too awesome and bloody fast :-). Also, Memcache can run in a clustered environment, so no need to manually flush the remote cache. The memcache Drupal module and memcache daemon would take care of it
• Because of the movement of caching to memcache, our databases weren’t on heavy load any longer. Those MySQL cache queries were really hard to process for our MySQL daemons. The MySQL BIN log files are growing now very slow, which is perfect!

MySQL HA

Both MySQL daemons are clustered using MySQL High-Availability (MHA).

MHA performs automating master failover and slave promotion with minimal downtime, usually within 10-30 seconds. MHA prevents replication consistency problems and saves on expenses of having to acquire additional servers. All this with zero performance degradation, no complexity (easy-to-install) and requiring no change to existing deployments. MHA also provides scheduled online master switching, safely changing the currently running master to a new master, within mere seconds (0.5-2 seconds) of downtime (blocking writes only). MHA provides the following functionality, and can be useful in many deployments in which high availability, data integrity and near non-stop master maintenance are required.

So how does it work?

• MySQL HA serves 1 virtual database IP. Both web instances connect to that IP. In the background, 2 database instances are running in MASTER-MASTER replication, but one instance is active, the other is passive. The difference is made on a percentage. For example: database 01 is receiving percentage 150%, the other one 100%. But what does that mean? It means all queries coming from both the web instances are going to the MySQL daemon with the highest percentage. The other (passive) one is idle (no requests are coming in), but the passive one keeps track of all changes of the active one (using the well-known MySQL replication mechanism). If the active one goes out (e.g.: too much CPU load, database unavailable, ...), the percentage on the active one drops from 150% to 100%. The percentage of the passive one increases from 100% to 150%. So all requests will now be sent to the other database instance.
• This MySQL HA has one disadvantage: there’s a single point of failure. If the virtual DB IP is unreachable, all traffic to the web intances will be gone - they cannot contact the database any longer. Also, database 01 and database 02 must be strong enough to capture the MySQL CRUD queries from both web instances. But as long as you keep caching queries out of MySQL, I do think you’re fine.

File storage replication

A last improvement would be to store all data on a NAS file storage.

• The NAS storage holds all data in sites/##YOUR_SITE_NAME##/files directory. Compared with the previous solution, we don’t need to sync data again. And we can delete files, nice! I would suggest to have the source code on both web instances, and not on the file storage. This makes it possible to release new "source code" (not database!) instances of Drupal modules. Or you can quickly change some lines on a PROD environment for debugging (as long as you block traffic from visitors to that web instance of course ;-)).
• Again: one disadvantage here: if the NAS file storage goes out: no file in your files will be served. Nor by webinstance 1, nor by webinstance 2.

So, happy to have written this down :-). It took me a while! I hope you enjoyed reading this blogpost. Let me know what you think about this new architecture.

Spotted @ #416 Trance Around The World with Above & Beyond

Spotted @ DJ Tiësto's Club Life - podcast 256!