We're WebVisionary award finalists!

Brian Jamison — Thu, 13 May 2010 01:48:26 +0000

We're delighted to hear that the Solar4RSchools.org project is a finalist for a WebVisionary award!

You can read more about the Solar4RSchools.org project here.

DrupalCon San Francisco, departing thoughts

Jonathan Hedstrom — Fri, 23 Apr 2010 19:41:38 +0000

I have just returned from DrupalCon San Francisco 2010, the largest DrupalCon ever. With over 3000 attending, and $72,000 worth of coffee consumed, it was a very lively event. Thanks to the awesome work of those organizing, nearly all of the sessions now have high quality video available on the DrupalCon site, and on Archive.org. Below are the sessions that really left an impression on me regarding the future direction of Drupal.

Advanced Drush

While there is no video for this as of this writing, the highlight here was the sudo drush make-me-a-sandwich moment (as well as the Drush core-cli command).

The Heart of Open Atrium: Context, PURL and Spaces

While we've been using context on most projects for quite some time, the work presented here around the spaces presets has me itching to try integrating them into some upcoming projects.

Page render drill down in Drupal 7

The page rendering system in Drupal 7 waits much later in the process to transform structured data into the relevant output format (which is typically HTML). Practically speaking, what this allows for is hook_page_alter, which has access to the contents of the entire page, in a structured array. Two highlights:

The block module will no longer be a required module in Drupal 7. Rather, it is simply an optional UI for placing blocks in the page. This means that the Context module in Drupal 7 won't have to implement weird workarounds to steal block placement from the block module (and as such, should work seamlessly with the i18n module).
Instead of only having to caching options (page and block level), any part of the page can be cached. Furthermore, by using the query string and qrguments as the cache key, these caching mechanisms can work with content that is under access control.

PubSubHubbub to the rescue-- Real-time feeds and the future of social networks

This talk wasn't specific to Drupal (yet anyway), but rather, provides a sort of roadmap to the future of Drupal and the social web. By combining 4 rather new protocols (PubSubHubbub, WebFinger, ActivityStreams and Salmon), Brett Slatkin of Google walks through the potential for realtime feeds being used to contect people like never before.

It would be very powerful if any Drupal site out there were capable of sharing, in real time, at the level Brett outlines in this talk. The Feeds module is already capable of consuming PubSubHubbub, and support for ActivityStreams have a little momentum. Salmon and WebFinger, as well as publishing to hubs for PubSubHubbub are the pieces that need to be filled in.

Drupal 7 and the State of Drupal

At Dries' keynote, he showed a best case scenario (June 2010), a worst case scenario (October 2010), and an ideal scenario (lock the 3000 attendees in until it was released at the conference) for the Drupal 7 release.

While we already missed the ideal scenario, the progress made on critical bugs at the conference has the community working at a velocity that should come in closer to the best case scenario. Over the course of the week, thanks to the awesome gatherings at the chx coder lounge, the number of critical bugs dropped from 114 to 94 as of this writing.

Enabling a multilingual global community for Mercy Corps with Drupal

Brian Jamison — Mon, 19 Apr 2010 23:33:40 +0000

We've just published a write-up of our work for Mercy Corps, where we enabled hundreds of youth organizers worldwide to join together in a global community, including multilingual support in Drupal. Read all about it here:
http://www.opensourcery.com/portfolio/projects/mercy-corps-globalcitizen...

How to write an RFP, part two: How much will it cost?

Brian Jamison — Wed, 03 Feb 2010 18:08:35 +0000

In my last post I talked about how important it is to include a budget in your request for proposal (RFP). People have since asked, “How do I know how much it will cost?”

I think that's the wrong question. There are two better questions. The first is, “how much will we get?” The second is, “how much will we save?”

How much will we get?
If your goal is concrete, say online fundraising or ecommerce, take the net funds currently raised after all costs for a year. Add your current growth rate and estimate that figure for the next year. Use real comparable data from similar efforts if you don't have your own.

Now multiply that by your goal. the amount of increased value you expect the site to deliver. Make your goals realistic and conservative. Perhaps your goal increased subscriptions of 20%. Your maximum budget ought to be less than the amount you expect to gain from the new project.

For example, let's say our site brought in $1,000,000 in 2010 with $250,000 net after all expenses. In 2009 that figure was $800,000, so we have a 25% growth rate. If that trend continues we'd expect $1,250,000 in 2011, or a net of $310,250. If our goal is to increase our subscriptions by an additional 20%, that gives us $370,500 in the first year. The difference, $370,500 - $310,250 gives us a maximum budget of $60,250. If our project costs less than that and delivers on the 20% goal, the project is a success.

Before I address our second question let's talk about budgeting when the goals are less measurable.

How much do we have?
In many cases the goal is less concrete. Awareness campaigns, political movements, and educational organizations fall into this category. In this case you need to know your budget before you go shopping.

Identifying a figure in a simple formula is impossible, but most organizations are putting the bulk of their outreach budget into online campaigns because it tends to be the most efficient use of budget. On top of that, results can be tracked and therefore easily improved, and the time to engage and close your prospect is faster.

No more than 50% of your budget should go into the entire technology deployment. At least 50% needs to be held back for staff to do the work – creating content, putting together online marketing campaigns, tracking the results and improving the campaign for the next run.

How much will we save?
This question is one of the most overlooked, least considered aspect of a technology buying decision. For example, a solid content management system (CMS) like Drupal can save an organization enormous amounts of time and money, particularly when compared to a proprietary closed CMS or a website managed by Dreamweaver.

The areas where savings can be had in a Drupal CMS fall into over a dozen categories, not all of which will apply to every organization. Those categories are software licensing, license management, content management, technical management, hardware, hosting, security/reliability, support contracts, development or customization, training, opportunity cost, hardware upgrades, software upgrades, and eventual migration to another system. Total up actual or projected costs for each of these areas. The difference between your existing costs over the expected lifetime of the CMS is what you stand to save (or lose). I'll cover each of the categories in more detail in future posts.

For example, your costs to manage a proprietary CMS might be $175,000 per year, and you project costs of $125,000 per year to manage Drupal so $50,000 a year stands to be saved. If you expect your new Drupal site to last you three years, that savings is $150,000. Coupled with our prior budget of $60,210 that gives a maximum budget of $210,250.

Answering these questions should help your organization identify a budget for your project.

Boosting performance on a permission system with nested roles.

Chad Granum — Tue, 29 Dec 2009 16:17:39 +0000

We discovered a problem while QA'ing emC against a huge client database. The page which displays who has access to a client would take a full minute to load. This was deemed unacceptable, and a solution was found.

The elementalclinic permissions system works on the idea that there are clients, personnel (staff members), roles to which personnel are members, and permissions for roles to access clients. Every staff member is given a 'primary role' that only they are a member of. Personnel become members of roles when their primary role is added as a member of a system role. This also allows for nested roles where a staff member is a member of system role a, which is a member of system role b.

Nested roles are useful when you want to share permissions between roles. For instance there is an all_clients role which grants access to all clients. The Admin role, being superuser, should also have access to all clients. With nested roles you simply make admin a member of all_clients. You can then add admins to admins, and non-admins who need all_client access to all_clients.

The problem with nested roles comes when you need to check if a staff member has permissions to view a specific client. You need to build a list of all roles the staff member is a direct or indirect member of. You then need to check if any of these roles grants access to the client. On the access page this needed to be done for each staff member.

To solve this problem I decided to leverage postgres. In the initial revision of the permissions system the database simply held the roles, direct memberships, and permissions. Traversing the roles and assessing client permissions for specific users was left entirely to the code. In the new revision I decided the database would maintain a complete map of memberships, both direct and nested. In addition the database would provide several views to simplify obtaining the results we need.

The focus of the changes is with the table that hold memberships. Initially it's schema was very simple (Old way):

 21 CREATE TABLE personnel_role_member(
 22     rec_id        SERIAL NOT NULL PRIMARY KEY,
 23     role_id       INTEGER REFERENCES personnel_role( rec_id ) NOT NULL,
 24     member_id INTEGER REFERENCES personnel_role( rec_id ) NOT NULL,
 25     -- Make sure duplicate memberships do not occur.
 26     UNIQUE( role_id, member_id ),
 27     -- Roles should not be members of themselves.
 28     CHECK( member_id != role_id )
 29 );

Here is the new schema for tracking role memberships:

 21 CREATE TABLE role_membership(
 22     rec_id         INTEGER UNIQUE NOT NULL PRIMARY KEY DEFAULT nextval('role_membership_rec_id_seq'),
 23     role_id        INTEGER NOT NULL REFERENCES personnel_role( rec_id ) ON DELETE CASCADE,
 24     member_id      INTEGER NOT NULL REFERENCES personnel_role( rec_id ) ON DELETE CASCADE,
 25     direct_cause   INTEGER REFERENCES role_membership( rec_id ) ON DELETE CASCADE,
 26     indirect_cause INTEGER REFERENCES role_membership( rec_id ) ON DELETE CASCADE,
 27     -- Roles should not be members of themselves.
 28     CHECK( member_id != role_id ),
 29     CHECK( direct_cause != indirect_cause ),
 30     -- Duplicates are pointless.
 31     UNIQUE( role_id, member_id, direct_cause, indirect_cause )
 32 );

The main thing to note in the new schema are the 'direct_cause' and 'indirect_cause' fields. These reference other entrees in the same table. The point of this change is to track both direct and indirect memberships in one table. When a membership is added a trigger fires which adds an entree for all resulting indirect memberships.

The trigger code is fairly complicated, so I am posting a simplified psudo-code version. This trigger is fired whenever an insert occurs on the role_membership table. Note: In this psudo-code 'us' or 'we' refers to the role that is becoming a member.

  Check for recursive memberships, raise an exception if found.
 
  Add memberships to all the roles the role we just became a member of is a member of. Direct cause is the new membership, indirect cause is the existing membership
 
  Add memberships to the role we just became a member of to the roles that are members of us. Direct cause is the existing membership, indirect cause is the new membership.

This trigger also fires off for any indirect memberships added by the trigger recursively. This trigger results in a table that always has a complete list of both direct and indirect memberships.

From this table it is a simple matter to create views for just direct or indirect members. Views for which roles have access to which clients is also fairly trivial. We can even create a view that shows client permissions with the reason(membership) that grants them.

Here are links to the original permissions schema, as well as the migration to the new system.
Original
Migration
And here is some additional documentation
README

Retrofitting a role and permissions system to a large application.

Chad Granum — Mon, 12 Oct 2009 17:48:49 +0000

The project

Recently I was given the task of overhauling the permissions system of eleMentalClinic. We needed a way to specify which clinicians could view which clients. We needed to ensure client security by preventing unauthorized clinicians from viewing certain clients.

Background

The original eMC permissions system was very simple. In the database there was a list of roles. There was also a table that listed client->role relations. Roles were identified by name. Roles were only used to verify a clinician had access to a given controller.

The plan

This refactor was a multi-step process. The first step was to write a new role+permissions system that worked with the controller-access logic, but also allowed for clinician+client associations. The second step was to retrofit security check logic to the application.

The first step is fairly simple. Creating roles and associations like what we needed for eMC is a common problem with several common solutions. I ended up migrating to a new system where we had system roles, personnel roles, role memberships, and role-client associations. Memberships are all role-role associations.

The problem

The second step is significantly more difficult, specially when you consider the size and complexity of eMC. One thought is to have the dispatcher intercept the client_id parameter and run a check against the current_user. The problem with this solution comes on pages where multiple clients are displayed, many without a request.

The next possibly solution is to modify the client object to require authorization in order to build. The problem here is that the current_user is known to the controller, but not globally. Making the current_user a global did not sound like a good idea, in addition this would break a majority of the tests in the eMC test suite, a suite that is 2.9mb. If we had to refactor the client object, and all the resulting test failures, this task could take months.

The solution

I began toying with the idea of code that could retrofit the permissions into eMC at runtime. A magical module that would be able to provide the security check with all the information it needs. A module that could ensure security without the need to rewrite any of eMC's existing object code. Enter Package-Watchdog. Package-Watchdog is a Perl module I wrote to do just that.

With Package-Watchdog you can 'watch' subroutines within a specific package, and 'forbid' them from using subroutines in another package. The way it works is simple in concept, wrap code around the watched sub that rewrites the forbidden subs. During normal use the forbidden subs act as normal, However once inside a watched sub the forbidden ones are altered. This continues all the way down the stack, you could have a chain of subs as deep as you want, if any of them call the forbidden one the watchdog is triggered.

By default Package-Watchdog will die when a forbidden sub is invoked within a watched sub. However Package-Watchdog is written so that you can provide a custom reaction subroutine. This custom reaction subroutine is passed all the information relevant to the watch. This includes the names of the subs called, the parameters they were called with, and the watchdog helper objects that manage them.

Package-Watchdog in eMC

In eMC a watchdog object is instantiated just before the controller. The watchdog is told to watch the controller's subroutines, and forbid the subroutines in the client object that are used to build it. The watchdog is provided a custom reaction sub. This custom reaction sub gets the controller via arguments, and the current user from the controller. The client_id being loaded is provided via the forbidden arguments. The forbidden sub can then take this information and run a security check. If the check passes the program continues, otherwise it throws a security exception.

The main goal of the watchdog is to absolutely prevent displaying client information to an unauthorized clinician. The watchdog left the application badly broken, pages such as schedule would always throw security exceptions. This is still considered a win, it means no accidental displaying of client information.Now it becomes a task of fixing the security exceptions, and as an added bonus we can now see where they are so they don't slip by.

As one might expect this could slow down an application considerably, and at first it did. I added caching to the permissions system, and also to the security check logic. The application now runs as fast as it always has in my experience.

Links

Package-Watchdog on CPAN
Package-Watchdog on github

Object#let in Ruby

Zack Hobson — Fri, 09 Oct 2009 15:25:26 +0000

A common construct in functional programming languages is the let macro, used to define lexically-scoped names. The let macro exists in most (all?) Lisp dialects and spiritual descendants, including Clojure. Since it's possible to define lexically-scoped variables at any time in Ruby, there isn't much perceived need for a construct like let. However, Ruby's block syntax makes it particularly easy to experiment with functional programming techniques, and in fact there is already a method in Ruby's standard library that is maddeningly similar to let, the tap method.

The reason I assert that tap is "maddeningly similar" to let is that tap does not return the result of the block, rather, it always returns the receiver. This makes it possible to take a value or values and create lexically-scoped names referring to them in a block, but there is no way to access the results of that block unless you explicitly stash them in the receiver or (possibly) a global. Even still, tap has its uses, which is why it's been in the standard library since 1.9.

# output: up is down
"down".tap { |up| puts "up is #{up}" }

A Ruby implementation of let would be similar to tap in that it would allow you to create a lexical block, but it would differ in that it evaluates to the block result:

# output: up is down
puts "down".let { |up| "up is #{up}" }

In cases where you don't care about the result of the block you can use either one, but if you care about the block result instead of the receiver then you can use let instead of tap:

# output: UP IS DOWN
puts "down".let { |up| "up is #{up}" }.upcase

I find myself wanting to use tap or let in cases when I need to use a calculated value multiple times. I could just create a temporary variable, of course:

c = some_complex_expression
do_something if c
other_method(c)

Instead I can use tap (or let if it's available) to achieve the same effect:

(some_complex_expression).tap do |c|
  do_something if c
  other_method(c)
end

This is quite useful in ERB templates, where creating temporary variables feels especially wrong.

You can also exploit a feature of Ruby to get multiple lexical names per block. If the receiver is an Array and you provide multiple names in the block arguments, the contents will automatically be broken out. This works with either tap or let:

(1..3).to_a.tap do |one, two, three|
  puts "one is #{one}"
  puts "two is #{two}"
  puts "three is #{three}"
end
 
puts (1..3).to_a.let { |one, two, three|
  "one is #{one}", "two is #{two}", "three is #{three}"
}.join("\n")

The implementation of both methods is trivial. It's not like any developer who wants to use let needs to wait for some Ruby core developer to code it up for him. Here's the complete implementation of Object#let:

class Object
  def let
    yield self
  end
end

The tap method (which is already in the standard library) is about twice as large, weighing in at two trivial lines:

class Object
  def tap
    yield self
    self
  end
end

To other developers that stumble across this, I pose this question: what techniques and/or idioms do you favor when a scoped temporary variable is called for?

Roll your own context conditions

Jonathan Hedstrom — Wed, 07 Oct 2009 22:39:54 +0000

The context module provides great flexibility in terms of the available ways to set a context, but it's also remarkably straightforward to define custom conditions.

In order to set arbitrary contexts, hook_context_conditions() is implemented to define values and such. This example will use the value of a CCK field, called field_foo:

/**
 * Implementation of hook_context_conditions().
 */
function mymodule_context_conditions() {
  $items = array();
  // The key used here will be used below when setting the context.
  $items['mymodule_foo'] = array(
    '#title' => t('Field Foo Value'),
    '#description' => t('Set this when field_foo has a certain value.'),
    '#options' => array(
      // Note, these are hardcoded here for simplicity, but could
      // easily use something like CCK's content_allowed_values() api
      // function for more dynamic population.
      'value1' => t('Label 1'),
      'value2' => t('Label 2'),
    ),
    '#type' => 'checkboxes',
  );
  return $items;
}

The only other step is to actually fire the code that sets the context somewhere. In the example of setting a context based on the value of a CCK field, this can be done in hook_nodeapi(). But something like hook_init() can also be used.

function mymodule_nodeapi(&$node, $op, $teaser, $page) {
  if (isset($node->field_foo[0]['value']) &&  $op == 'view' && $page && menu_get_object() === $node) {
    // Use the same key here as used to define the context condition
    // above.  Note, this logic would need some re-working if
    // field_foo allowed multiple values.
    context_set_by_condition('mymodule_foo', check_plain($node->field_foo[0]['value']));
  }
}

That's it. Contexts can now be added through the UI or through code that react to the value of field_foo.

How to write an RFP, part one: Always include a budget

Brian Jamison — Mon, 21 Sep 2009 19:37:57 +0000

I'm writing this series of blog posts to help improve the response rate on your Requests for Proposals (RFPs) as well as increasing the chance that the right firms will respond. We receive a number of invitations to respond to RFPs every week, and a surprising number do not contain enough information for us to respond.

Rule number one, always include a budget in your RFP. You may be thinking that disclosing your budget is a mistake, that it leaves you open to predatory firms, or price-gouging. I disagree, and I'll tell you why.

RFPs without a budget are perceived as risky to developers. We wonder -- does the requestor have money set aside for the project? Is the project even approved? Is it in too early a stage for the developer to get involved? Does the requestor understand the complexity required?

Perhaps more importantly, a developer needs to know that they're the right fit. Because of our size, it doesn't make sense for OpenSourcery to respond to an RFP if the requestor has a budget of $2,500, and it also doesn't make sense for use to respond to a budget of $5,000,000. It could be argued that as developers we should know how much a given project will cost. That may be so, but finding the answer is time consuming and therefore expensive. Providing that answer is valuable; it requires expertise. Don't expect that to be provided for free -- at least, not by people capable of giving an accurate answer.

Responding to an RFP has real costs. A developer must dedicate expensive resources; for example it costs my company at least five hundred dollars to respond to an RFP, both in hard and opportunity costs. It can cost us many thousands. I need to be convinced that there's enough profit potential in an RFP to justify the investment, and without a concrete budget I'm forced to guess. And my guess will always be "not enough."

Adding a budget is the most important component to an RFP. Don't leave it out.

If you don't know how much your project might cost, you need to find that out before you put your RFP out to developers. I'll cover techniques for doing that in my next post.

Achieving success, part three: Stop re-inventing the wheel

Brian Jamison — Fri, 11 Sep 2009 00:46:51 +0000

One of the major reasons I believe in open source is efficiency. Leveraging open source software should allow you to spend your time and money on customization or innovative new features.

For example, it doesn't make sense to build a content management system from scratch when you can leverage the efforts of hundreds of thousands of programmer hours for free. And yet, developers do this all the time.

Open source developers aren't exempt. And simply using open source tools doesn't automatically guarantee the wheel isn't wastefully being re-invented. You have to leverage an open source project to do that.

What's the difference between tools and projects?

Tools are just what they sound like; they allow you to build, but they're not the end product. PHP and Ruby on Rails are tools. Projects are finished work and in many cases need nothing more than installation to be useful. Drupal and Wordpress are examples of open source projects.

Developers often say, "Don't worry, I'm using open source tools on this project!" and happily waste hundreds of hours duplicating work that is freely available.

Why would a developer re-invent the wheel? I don't have a good answer for that, but I can speculate. They're probably not trying to make a project cost more. They may underestimate the difficulty, or the complexity involved. They might not be looking out for your best interests; who will manage the site after they move on, or how will you be able to inexpensively add new features down the road. They may be unwilling to invest the the hundreds of hours mastering an open source project. Or they may believe they can do it better; they have a grand vision for turning your contract work into a new open source project. No problem -- unless you, the client, have to pay for it.

I have three recent examples. The names have been changed to protect the re-inventors, but I've encountered each within the last two months.

Case #1
We were recently asked to evaluate a content management system written from scratch using the open source Ruby on Rails tools. And while the site looks beautiful, they wasted time and money re-inventing the wheel. Instead of spending months building the site from scratch, they could have installed Drupal in minutes and invested in useful new features, even better design, or just saved the money.

Case #2
A company spent tens of thousands of dollars customizing Drupal to great benefit. They attributed much of the success of their product to the marketing and social functions their site gave them -- features other competitors lack. Unfortunately they recently hired a junior programmer. The junior programmer threw the old site away and spent months building a new website from scratch using open source tools, but not re-using any of the prior work. And while the new site looks great, the old site could have been themed to look exactly the same as the new site in about two days, retaining the sophistication already developed. As it stands, who will maintain this site once the junior programmer moves on? Are there unknown but serious security holes in the new system?

Case #3
This week I met a web developer who built a new content management system for a client from scratch using PHP. He wondered whether or not it would be a good business opportunity to sell it to other companies. I had to tell him that there is simply no way that a new content management system written by two guys will be able to compete in the overcrowded content management space. Their fresh new system lacks the thousands of free enhancements available, the sophisticated permissions, workflows, solid security and design flexibility that Drupal offers. And it will never be able to catch up, because hundreds of people are working every day to make Drupal better.

In each of these cases, tens of thousands of dollars could have been saved or put to better use. Please don't re-invent the wheel. Hold your developers accountable to their design decisions. Ask them how they intend to leverage existing open source projects to save you money.

In future blog posts I'll talk about productizing applications in a niche market, but beware the developer that talks about turning your consulting job into a mainstream product like a content management system.