Jonathan Hedstrom's blog

On a recent project involving CiviCRM, I really needed to write tests for the workflow, which was very complex, and without tests, brittle.

The solution, partially, is Drupal's SimpleTest (which now uses it's own framework, instead of the original SimpleTest library). I say partially, because unfortunately, CiviCRM sits completely outside of the Drupal API, and doesn't play well with things such as hook_install(), which the Drupal testing framework uses to build a parallel testing database each time tests are run.

As such, the mostly complete solution was to use my custom workflow module's setUp() method to copy the existing CiviCRM database, and then run tests against that:

  /**
   * Implementation of setUp().
   */
  function setUp() {
    parent::setUp('civicrm', 'os_custom', 'os_civicompany');
 
    // Now, clone civicrm install
    global $db_prefix;
    $test_db = db_set_active('civicrm');
    // Build list of tables
    // @todo - implement this for postgresql
    $result = db_query("SHOW TABLES");
    while ($table = db_result($result)) {
      // Build list
      $this->tables[] = $table;
    }
 
    // Clone tables with prefix
    foreach ($this->tables as $table) {
      // Note, we intentionally DON'T use the bracketed table names here,
      // since the whole point is to create a set of tables that will work
      // with the testing $db_prefix.
      db_query("CREATE TABLE " . $db_prefix . $table . " AS SELECT * FROM " . $table);
    }
    db_set_active($test_db);
  }

This set of code first runs the parent setUp() method, which creates a new Drupal instance inside the existing database, but with all the tables prefixed with the $db_prefix. It then switches to the CiviCRM database, and copies each table into an identical table, but named with the testing prefix. On advantage, that is mostly a side-effect, is that instead of having a fresh install of a CiviCRM database, I get a testing copy that is configured exactly the same way as the application I am trying to test in the first place^*.

In order to clean up after the test, a custom tearDown() method is also needed:

 /**
   * Implementation of tearDown().
   */
  function tearDown() {
    // Cleanup CiviCRM database.
    global $db_prefix;
    if ($db_prefix && ($db_prefix != $this->db_prefix_original)) {
      // Only drop tables if $db_prefix is set, and not equal to the original
      $test_db = db_set_active('civicrm');
      foreach ($this->tables as $table) {
        db_query("DROP TABLE " . $db_prefix . $table);
      }
      db_set_active($test_db);
    }
 
    parent::tearDown();
  }

This simply goes and removes the prefixed copy of the CiviCRM database.

These two methods allow for my custom Drupal modules to test against the CiviCRM database in most cases. Where this method fails is for testing actual calls to the CiviCRM API. Since this API sits completely outside of Drupal, it isn't fooled by simply switching the global $db_prefix for the tests. It may be possible to trick CiviCRM into operating on the test database, but as of this writing, I haven't found an elegant way to do so.

* A similar method could be used to duplicate the configured Drupal application. Instead of calling parent::setUp(), the method would duplicate everything that function does except run the install, and instead copy every table into the test database.

I made it back from Hungary last Friday, and am only now starting to convince my body that it really isn't 9 hours later than it is. After Drupalcon, I managed to make it out to Eger, for castles and a valley of wine cellars (thanks for the recommendation Chrys). Following Eger I returned to Budapest to explore more castles, a Cold-war-era statue park, and some labyrinths all made possible by some of the best public transit I've experienced.

Being back, and having had a bit of time to process all that I took in at Drupalcon, I wanted to do a bit of a review of some of the highlights. For most of the talks I mention below, there are either slides, video or both available here.

The venue and organization

The venue itself was very nice. Spacious and airy, it was the perfect place for Drupal geeks from all over the world to congregate and talk about, well, Drupal. Also, many thanks to all the volunteers and the Hungary Drupal community that made Szeged a reality.

I can't talk about the venue and the organization of the event without at least mentioning the Hitchhiker's Guide to the Galaxy theme. I won't go into too much detail, as the pictures do that, but suffice to say I thought it was awesome and thorough (42+ languages, a babelfish, don't panic and the certainty of knowing where one's towel is).

The State of Drupal

Dries, while opening the conference with his State of Drupal talk, hit on many aspects of the future of Drupal.  One highlight was the idea of embracing RDF, which not only makes for richer content, but addresses the problem of importing and exporting for Drupal in an application-independent manner. Another very interesting point was that most Drupal developers feel they are best at HTML, and by the same token, want to learn about HTML the least. That's good I suppose...

Drupal 7 preview

Many of the sessions I attended provided a very exciting preview of Drupal 7, and I came away with high hopes for the new version.

Larry Garfield proposed the idea of getting handlers into core. My favorite example of why this might be useful was about the idea of swapping out default file handling, for say, a file handler that sends all images to Flickr, and all videos to YouTube, but behaves normally for everything else. To do this without handlers, one would need to either hack core (oh noes), hack image and video modules, or write a custom module. With handlers, the modules can simply be concerned with the Drupal API, thus making this sort of behavior module-independent. Essentially, handlers are one of the last steps towards making Drupal completely customizable without hacking core (thus saving kittens).

The very next day, Larry gave an overview of the new database layer that landed, which takes advantage of PHP's PDO abstraction layer, shortly before Drupalcon. The short of it is that the new abstraction layer looks awesome. The dynamic query builder will take the place of complicated string concatenation and pattern matching that plagues certain hooks and modules today. The new layer also provides—out of the box, so-to-speak—the ability to support master/slave replication and transactions, to name a few. Also, since this is all object-oriented, simple (but fragile) INSERT statements are no longer directly coded into modules. Instead, inserts and updates are built up using placeholders, and can be reused. Finally, the new db_merge() function will make checking for an existing record (in order to determine if an INSERT or an UPDATE is needed) a thing of the past.

Barry Jaspan gave a talk on the Field API that will hopefully be landing in Drupal core for 7 (ie, CCK in core). However, the field API isn't just about moving the CCK module's functionality into Drupal. Along the way, the code will be transformed into a real API (so that the creation and modification of fields is no longer dependant on a form submission). Another key change is the idea that fields will be attachable to anything, not just nodes. During the talk, field definitions looked to be custom classes, which seemed very clean and extendible. It was later rumored around the halls that this idea had been scrapped, but I'm unable to find any hard evidence indicating one way or another.

Peter Wolanin discussed the idea of page rendering being capable of outputting many different formats in Drupal 7. The central idea of the talk was to move page rendering to a later stage in the process, so that data can be rendered as, for example, JSON, XML or XHTML, depending on any number of factors. The only way to currently change format, is to either pre-empt the rendering level (by exiting the script early) or to re-render data, which is inefficient at best. Needless to say, this would be a welcome change.

And finally, the new file handling patch continues to get closer to RTBC.

So long, and thanks for all the fish

That's all I have time for now, although there were many other very exciting and informative sessions/BoFs (I saw a demo of the progress being made with the Geo module, as well as the work Development Seed is doing with Mapnik, and am very excited about the future of mapping and Drupal).

Today, on the first day of Drupalcon, Rasmus Lerdorf gave a fantastic presentation about PHP, performance, scalability and security. In this talk (the slides are available here), he discussed the writing of frameworks, and how they tend to trade off performance for what might be deemed as cleaner code. After walking through some very cool benchmarking methods (see slides 8, 14 and 16), he proceeded to benchmark some of the more popular PHP frameworks out there (slides 24 through 32). Unfortunately, when it came to Drupal, it was benchmarked without caching enabled. I have duplicated his Hello World module, and enabled it on a clean Drupal 6 install. While the absolute numbers are incomparible due to my local environment being different from his, I have run it without caching

Response time:                  0.38 secs
Transaction rate:              13.10 trans/sec

and then with aggressive caching enabled:

Response time:                  0.04 secs
Transaction rate:             119.42 trans/sec

The performance increases by roughly 9. By applying this ratio to his results for Drupal:

Response time:              0.10 secs
Transaction rate:          51.37 trans/sec

the numbers would be something like:

Response time:             0.0105 secs
Transaction rate:           468.5 trans/sec

With the pure html page coming in at about 610 transactions per second, Drupal 6.4 with aggressive caching enabled (as well as css and javascript optimization enabled) is much closer to that than any of the other applications/frameworks tested for the presentation. Of course, not having the exact same environment as Rasmus, I'm not sure if these results scale linearly, so this is mostly speculative. Even so, it takes Drupal a lot closer to that goal of not killing kittens, and a greener computing world.

Edit:

The above numbers were without APC enabled. With APC, the ratio is more like 5-6, so in that case, Rasmus' numbers would look like this:

Response time:              0.017 secs
Transaction rate:          256.85 trans/sec

While not as good as those above, still these numbers come in very close to the best of the applications benchmarked by Rasmus.

Tomorrow I fly to Budapest by way of Frankfurt as I start making my way towards Drupalcon in Szeged. Brian will be joining me in Szeged closer to the start of the conference. I'll have several days to take in the sites of Budapest (the funicular railway to Buda Castle looks awesome), then catch a train south to Szeged. I'm looking forward to meeting so many Drupal people all in one place, and the sessions are looking fantastic.

I'm particularly excited about the Awesome Testing Party, not only due to the free pancakes, but also, writing tests is a whole lot of fun, and they're critical for a solid Drupal core, and a more feature-rich development cycle.  It also looks like there should be several excellent sessions on the state of GIS, mapping and Drupal (A Roadmap for Mapping: GIS on Drupal in 2008 and Beyond, Simple Mapping Mashups with Drupal, and Mapping with Drupal and Mapnik just to name a few). And of course, no Drupal gathering would be complete without a session about Awesomeness and Drupal.

I'll be posting updates frequently from Drupalcon. Also, if you'll be in Budapest prior to Drupalcon and are interested in any pre-Drupalcon activities, drop me a comment here, or contact me directly.

I recently had to transfer some tables from a Drupal site that had been developed in it's own database, into an existing Drupal database. In order to do this, the MySQL dump needed to be edited to add table prefixes, lest the existing Drupal install be wiped out.

Editing this by hand was out of the question, but thanks to the emacs replace-regexp command, I was able to change the entire dump file in a matter of minutes.

To initiate the command, one hits ALT-x and types in replace-regexp. First, enter the pattern to find. To add a prefix to all the DROP TABLE commands:

DROP TABLE IF EXISTS `\(.*\)`

The \(.*\) is the regular expression that tells emacs to match everything inside those parenthesis.

After entering the search, one is prompted to enter the replacement pattern. To add a prefix, for example myprefix to the DROP TABLE commands I used this pattern:

DROP TABLE IF EXISTS `myprefix_\1`

The \1 pulls in whatever was matched in the first set of parenthesis in our search pattern.

I repeated a similar search and replace for the other commands that reference a table in a typical Drupal database:

 CREATE TABLE `\(.*\)`
 CREATE TABLE `myprefix_\1`
 
 LOCK TABLES `\(.*\)`
 LOCK TABLES `myprefix_\1`
 
 ALTER TABLE `\(.*\)`
 ALTER TABLE `myprefix_\1`
 
 INSERT INTO `\(.*\)`
 INSERT INTO `myprefix_\1`

After that I was able to import this dump file into the existing Drupal database (without wiping out the other installation) and continue development as a multi-site sharing a single database.

Last Friday I met Aaron and drewish at the Ace hotel for a day of SimpleTest writing for the long-awaited hook_file() patch. This was part of the Portland Media/Files Code Sprint, organized by drewish to solidify the patch during the week of OSCON.

The day was very productive, as file.inc went from having virtually no tests, to near complete test coverage. As is often the case when writing comprehensive unit tests, several inconsistencies with expected behavior/documentation were found and fixed.

Aaron has provided an overview of the goals of the code sprint, and the remaining work to be done (both on the immediate patch, and beyond). But to summarize, the goal is better file (thus, media) handling in core.

If you maintain or develop modules (or themes) in Drupal that need to deal with any sort of files, go give this patch a try (it's currently the spotlight patch).

I've had a tremendous amount of fun writing tests for file.inc (hopefully I'll get to do a few more before the patch goes in), so I may very soon jump on some of the other needed tests for Drupal core.

While working on a CiviCRMCiviCRM is an open source and freely downloadable constituent relationship management solution. CiviCRM is web-based, open source, internationalized, and designed specifically to meet the needs of advocacy, non-profit and non-governmental groups. site, I ran into a problem of migrating the installation from the development host to the QA, and from there to production. Since CiviCRM stores so much host-specific information in the database, every transfer needed to re-configure the host information. After much research and a little bit of pain, we discovered that the proper way to do this is to simply empty the config_backend field in the civicrm_domain table.

So with this little bit of code tacked onto our deployment scripts:

#Reconfigure the CiviCRM backend
echo 'UPDATE civicrm_domain SET config_backend = NULL WHERE id = 1;' \
| ssh $USER@$HOST "mysql -h $DB_HOST -u $DB_USER -p$DB_PASSWORD $DB_NAME2"

deployments are once again flowing smoothly.

While working on importing a bunch of content from an old Drupal site to a new one, I discovered a little gotcha with the D6 database schema functionality. The method I was using involved loading a bunch of nodes from the old database via a call to db_set_active(), followed by a series of calls to node_load(). Once I had the nodes in an array, I switch to the new db via another call to db_set_active and store the nodes via node_save().

The problem arose due to the fact that Drupal is caching database schema information. Thus, during the node load phase, CCK attempts to load up information on a table that doesn't exist, and the result is cached. When back in the new database, where the table really does exist, the cached schema thinks it doesn't.

The solution was to clear the database schema cache every time I switched back to a different database:

function import_old_set_db($db = 'default') {
  db_set_active($db);
 
  // clear schema
  drupal_get_schema(NULL, TRUE);
}

The key in the above function, which I used as a wrapper to db_set_active() in my import script, is the call to the drupal_get_schema() function, which clears the cached database schema information.

This was a viable solution because this is a one-time import script where performance really isn't an issue. It would not be recommended for production sites when regularly switching between databases during normal page loads.

I ran into the following issue while developing a bilingual (English and Spanish) site recently:

I wanted the site, for ease of use, to have a single menu structure so translators didn't have to worry about mimicking the primary navigation every time they translated a page or added a new one.

The problems, when the site was in Spanish, were three-fold:

1. Primary links were linking to es/english-path even though es/spanish-path existed.
2. For every sub item that was in English, when on the translated page, the secondary menu didn't display, since the menu system didn't see the translated page as a child of the parent.
3. Both primary and secondary link text was still in English.

The 3 issues were solved as follows.

in theme_preprocess_page:

  $vars['primary_links']   = _opensourcery_primary_links($vars['primary_links']);
  $vars['secondary_links'] = _opensourcery_secondary_links($vars['secondary_links']);

and these 2 functions look like this:

function _opensourcery_primary_links($primary) {
  global $language;
 
  foreach ($primary as $lid => $link) {
    $link = opensourcery_translate_translate_path($link);
    $primary[$lid] = $link;
  }
  return $primary;
}
 
function _opensourcery_secondary_links($secondary) {
  global $language;
 
  // This function call will rebuild the secondary menu as if the page were in
  // English, thus solving the second issue.
  if ($language->language == 'es') {
    $secondary = _opensourcery_rebuild_secondary_links();
  }
 
  foreach ($secondary as $lid => $link) {
    $link = opensourcery_translate_translate_path($link);
    $primary[$lid] = $link;
  }
  return $secondary;
}

As you can see, the key to each of these functions is the opensourcery_translate_translate_path($link) function, which looks like this:

The Session Favorites module provides a "Favorites" or bookmark system based solely on a browser cookie. This allows site visitors to aggregate lists of content without becoming full-fledged users.

The module can be configured on a per-node-type basis, and optionally provides AJAX add/remove links.

This module will soon work alongside the Favorite Nodes module, or possibly Views Bookmark (or both, depending on demand), such that once a visitor registers or logs in, their collected session favorites are transferred to the more permanent Favorite Nodes or Views Bookmark storage.

Whilst writing the Video Upload module, I came across Drupal's hook_requirements() function. As it turns out, this is the perfect place to check for all the little stuff your module requires in order to function properly.

For example, the video upload module won't work at all unless it has a YouTube developer key, a username and a password configured.

The Video Upload module, while in alpha state, provides a CCK field type that allows for the end user to upload video directly to YouTube, using a single account for the site.

The video never hits the Drupal host, saving on storage and bandwidth bottlenecks, and the end-user doesn't need a YouTube account, since all video is stored under the site's account. Video can be organized on YouTube with customized developer tags, currently with limited token support.

The module uses the Zend GData client library for communication with YouTube. This can be downloaded here. See Video Upload's INSTALL.txt for details on installing this library.

The module in extreme alpha state, with nightly builds available here.