Come the industrial revolution however, there was suddenly an issue of scale introduced into the equation.  Suddenly instead of a handful of customers, an enterprise might have hundreds, thousands or, in the modern world, millions and billions.  One of the first and most fundamental actions of the industrial revolution was to cut personalization out at its heart.  That’s because personalization didn’t scale.  The factory needed to get thousands of units manufactured on their assembly lines, and couldn’t stop to deal with pesky user requests like “colors-other-than-black” on their Model Ts.

Then cometh the Internet.  With its oh-so-seductive lack of physicality and supposed promise of infinite supply (and the associated supposed-zero-incremental-customization cost) there was suddenly a burst of activity around creating customized experiences for individuals on the web.  Personalization was back.  All sales leads would convert and prosper.

Wait a second, though.  Go look for some books on personalization.  You might notice something interesting, as I did when I went looking.  Nothing has really been published about personalization since about 2002 or so.  That’s kind of funny.  Wonder why that is?  Almost like…something went wrong…

Black Boxes

New technology always debuts as a black box that promises to solve all of our problems.  Generally speaking, the people signing the checks don’t care to understand how it works, only that it’s going to bequeath competitive advantage on them (in the case of early adopters) or because all of their peers are signing on to it (in the case of the majority).  Also, sadly, the role of technology has been treated in a very arms-length manner within much of the business world.  That’s too bad, because it has some of the greatest impact on business.

In the case of personalization, engagements typically went like this: you fed all of your data into it (products, articles etc) and let it perform some sort of proprietary analysis on that data.  Then you dropped some recommendation widgets onto your site, clicked your heels together three times, and suddenly you were looking at theoretically relevant content and/or personalized product recommendations.

Over time, however there have been concerns by people who have deployed such systems.  In a nutshell those concerns are:

• How do we know our recommendations are working?
• How do make sure the black box personalization system doesn’t do something weird or inappropriate?
• How can we use personalization to distinguish ourselves from our competition?

In the world of black box personalization, there haven’t really been good answers to these questions, because you’re dealing with a black box.  You don’t know how the system is really working, there aren’t any controls or visible workings that you can have mastery over and it’s difficult to gain insight on how things work, and how your business strategy could be incorporated into the personalization system’s execution.

I believe these combined factors took a lot of the sheen off of the face of personalization.  Business operators, unable to really get any idea about how these systems worked, decided they weren’t that happy about delegating such a fundamental aspect of their marketing to an automated system that provided no means of understanding its workings.  From there the only practical choice was to go back to manual curation.  And that, as we know, doesn’t scale.

This problem is, I believe, at the heart of why I’ve had trouble finding any writing about personalization that has been published within the last 5 years.  I have a hard time believing that the technology behind personalization has come to a halt, but the business application of it seems to have hit a brick wall.

Splitting the Dilemna

So faced between a choice between the scaling problems of manual curation, and the unpredictability and the unknown factor that comes with black box personalization, business hasn’t been left with a great choice.  Generally speaking, it seems like businesses have mostly come down on the side of manual curation, and have put personalization into a ghetto on their website (a recommendations sidebar or such).  Again, that’s too bad, because there’s so much being left on the table that personalization could address, but without more control and insight into the process, business are unable to trust it.

What would be nice is a mechanism for businesses to be able to define the principles, or rules, under which they want to extend personalized service, and then from those rules deploy automation to execute that service.  A kind of  ”white box” personalization, if you will.  This is where Conversational Architecture comes in.

I propose that we treat personalization a little differently than we have in the past.  Using Conversational Architectural terms, business can define how their personalization system should respond to particular cases that are important to the business, in the way the business prefers.  Because the business is essentially defining rules, however, and not individual cases, the system can scale through automation.

Tuning the Content Mix

Conversational Architecture thinks of product recommendations, or related or personalized content, as all part of the same problem.  We call these “content policy” objects, and treat them as components that can be included in various contexts.  A content policy object is a placeholder for a set of business rules that describes how a component should be populated with content (content meaning articles, products, profiles or whatever).  The point, however, is that the mix of content within these policy objects can be tuned by the website operator.

Here’s what kind of content you can get in a content policy object:

• Personalized:  Based on the user’s expressed affinities, personalized content tends to be related (via metadata) to content that the user has endorsed in some way.  For example, if we’re talking about products, then personalized products would be related, (via metadata, tagging or collaborative filtering style aggregation), to products in which the user has expressed an interest in some way, such as by buying them, adding them to a wishlist or so forth.
• Related:  Related content is similar to personalized content, but instead of user affinities, the starting point is something on the page that serves as a base content.  In the case of e-commerce, the starting point could be the product featured on a product page, or perhaps a particular metadata or tag value represented by a product category page.
• Curated:  Sometimes you just want some specific content to show up in a particular place.  This is curation, meaning the individual content objects are specified directly.
• Random: With all this content targeting, it’s occasionally nice to dial in a little bit of noise.  Random content could be anything, and allows the policy editor to back off a little from complete content targeting, allowing for the possibility of serendipitous discovery by users.  You never know.

How Content is Related

Another important idea with regards to personalized and related content is the idea of vectors of relevancy.   This has to do with the specific metadata that is used to relate content to each other, and the relative importance of similar values within that metadata.  Not all characteristics of content are equally important when tuning a content policy.

For example when we built Whiskey Engine (whiskey-engine.com), we consulted with an expert who explained to us the elements of the taste of whiskey.  He broke the taste of whiskey into elements like “herbs/spice”, “flowers” and “smoothness”.  Based on his tastings, we accumulated data about the taste profiles of each whiskey in his collection, with a value between 1 and 5 (1 meaning the least, 5 the most) for each taste characteristic.

However when we spoke with him further, it turned out that people tended to place a premium on the “smoothness” factor above all else.  In other words, once people decide how smooth they like their whiskey, they tend to follow that vector when looking for new whiskies.  People place far less concern on other whiskey taste values such as “herbs/spice”.

For a different example, think about music: if you like the artist who recorded one track, there are reasonably good odds that you’ll like another track by the same artist.  On the other hand, the record company that release the recording is much less significant.  People usually don’t care (and don’t know) which record company released a particular recording.

Buy carefully defining the metadata for content as well as specifying its importance as a vector of relevancy, recommendations and related content can be improved within policy objects.

Gaining Confidence: Measuring Content Policy Performance

So how can you know if your personalization strategy is working?  By treating the content policy object as the head end of a conversion funnel, and measuring how well it performs (we call this “Confidence“).  At its simplest, you should simply be able to see how well each content policy object is performing.  If you’re not satisfied with the results, then go tweak your content policy mix, or try altering the relevancy vectors of the content metadata to increase the quality of the personalized and related content in the policy.  Specific personalization deployments should never be treated as “set and forget”.  You need to work on them, adapting them to your specific audience.

Of course the ultimate version of responding to confidence measurements would be for the content policy object to reconfigure dynamically based on performance.  Conversational architecture has a name for this too – we call it “conductivity“.  Essentially we measure the performance of recommendations and weight the effectiveness of the current vectors of relevancy.  Over time we’ll start to adjust the vectors (“oh look, it turns out that herbs/spice is a bigger deal than we thought”) to increase the performance of the content policy objects.

What about Modality?

In the original conversational architecture article I talked a fair amount about modal websites.  How does modality apply to personalization then?  I think that first one should realize that while you don’t have to use modality to build a personalized website using conversational architecture concepts, modality can provide you with a way of describe some special features that could take your personalized website over the top.

The basic guideline is: when moving away from ideas that can be expressed easily in terms of content policy objects – modality can be very helpful.  For example, imagine an e-commerce website that provided discounts for people who frequently shared products from the site through social networking channels like Facebook and Twitter.  In Conversational Architecture terms, you could identify this kind of person as a Persona (call them “social sharer” or something) and trigger a mode from that persona that caused a discount to appear in their shopping cart.  This is an easily expressed business rule that could then be implemented through automation on the site.

Wrap Up

So we introduced five new concepts that Conversational Architecture provides that allow businesses to more clearly articulate what they want from personalization: content policy objects that provide customized mixes of content, products etc., vectors of relevancy that determine the relative importance of content metadata when calculating recommendations, confidence which expresses the performance of content policy objects in terms of conversion, conductivity which expresses the evolution of vectors of relevancy over time, and finally how modality can be applied to a personalization problem.

This is obviously a lot of new ideas, however it’s entirely possible to just try them out one at a time and look for incremental improvement in your personalization implementation.  By using these concepts, you have acquired a language in which to discuss personalization, which will allow businesses to more closely align personalization with their specific approaches to operating their business.

In part one we learned about Client Risk: risk that clients bring when they walk through the door.  Based on attributes such as the ability to make decisions, relative insensitivity to price and technology agnosticism, clients can either be part of the problem or part of the solution when it comes to managing risk.

In part two we learned about Structural Risk: risk that is derived from how the team is structured – whether they are familiar with each other, a common process, the tech or the client.  In some ways this type of risk is learning curve risk, because it models how many learning curves are piled onto the project.

Finally, in part three we learned about Qualitative Risk: risk that is driven from the quality of the team, the technology and the support for the team.  The better these things are, the more curve balls they can handle, and the more they can cope with risk from the other vectors.

From each of these three aspects of risk we derived an overall letter grade: from A to F.

Putting it All Together

Here’s how the letter grades combine:

Low Risk: All A’s and B’s

This is a good to great project with minimal risk.  In this range you can count on projects staying on track, and arriving where they are supposed to within budget, on schedule and at a level of quality that will satisfy everyone.

Medium Risk: One or Two C’s

This is a project that has a combination of a relatively problematic client, an immature team, or mediocre quality issues.  It is possible to succeed in this space, but it requires a close watch and quick action to mitigate the problems that inevitably will come up.  The success of the project depends on at least on of the three areas being better than the “C” level – the strength that can hold up the project.

High Risk: Three C’s or One or Two D’s

This is a project that, from its onset, has a good chance of going bad.  Either there is no strong area that pulls the project up to the medium risk level, or there is one or more areas with real problems: a truly high risk client, an immature team or poor quality from the team, the technology or the team’s support system.

Despite the best risk-mitigating activity from management, there is still a good chance that this project will go wrong.  It would be  a sound decision to walk away from such a project, but if taking it on is necessary, then it should be approached extremely defensively, as there will inevitably be serious crises as the project progresses.

Lead Balloon: One or more F’s, or 3 D’s

This is either a project with an impossible client, a disastrous team, or a catastrophic quality issue; or it is a project that is merely poor in all three areas.

Generally speaking, it is not possible to be successful with such a project.  The presence of the F will sink the project on its own, or the general malaise of the 3 D’s will accumulate into a disaster.

A project at this level needs a fundamental reset: generally either the client or the team has to go, depending on the source of the problems.  Risk mitigation at this level will be overwhelmed.

Conclusion

So there you go – my general theory of project risk.  I hope that you find it helpful, and through it gain a greater understanding of why projects succeed and fail, and when to plunge ahead, and when to run the other way.  Good luck!

In Part 1 we discussed client driven risk – the problems that clients bring to the table and how they affect the project.  In Part 2 we discussed structural risk – risk that originates with the composition of the team working on the project.

The third and list type of risk I call Qualitative Risk.  This is a kind of risk that is affected by the quality of the tools and people executing the project.  Everything counts, everything matters.  Quality affects the project.

There is a school of thought in business that would dearly like to factor quality out of the equation.  Businesses that trade in commoditized markets, competing on price, basically operate on the supposition that there is a standard level of quality and that it is not a distinguishing characteristic.  The movement towards off-shoring software development included an attempt to factor out quality.  Command-and-control programming languages like Java and .NET are attempts to engineer developer quality out of the equation – at least partially.

Quality, however, matters.  The web development framework you choose will affect the project, positively or negatively.  The skills of the developers you hire will affect the project.  It is not just subjective, some of these things are actually better than others.

I break Quality factors down into 3 main pillars: Team, Tech and Support.

Team: How strong is the team?  My feeling is there should be at least 1 really good senior developer for every 3 medium or junior developers on staff.  In addition, it is good to have at least a couple of honest-to-goodness rock stars around – the people who can be thrown against any technical challenge and still land on their feet.  The bar should be set at a level where the team can be counted on to astonish you on at least a semi-regular basis.

Tech: How good is the tech on which you’re building this thing?  Does it provide features that you need?  Does it provide a means of accelerating custom development, in case you need to go outside those features?  How often is time spent using the technology solving actual business problems, as opposed to just servicing the demanded ceremonies of the technology?  When is the technology enabling you, and when are you fighting it?

Support: The infrastructure: managerial, administrative, legal, HR etc., that surrounds the team – how much is it helping?  Is HR effective in finding new talent?  How about support from IT?  Is it straightforward to get ticket systems, continuous integration servers, new virtual servers etc. set up?  Or does every request take forever, with the development team resorting to circumventing the company infrastructure?  Does management back and support the development team, or are they selling them up the river?

Each of these pillars is deep and complicated, so grading them is a little abstract.  However if you set an aggregate grade for each pillar and take the average you’ll get a sense of the qualitative risk associated with the project.  Again, starting about the C grade and intensifying as you drop below that, you can easily get to a level of unacceptable risk.

In the next and final installment of this series – we’ll look at putting all the various risk factors, client-driven, structural and qualitative, into an overall risk profile.

I like to call this second category Structural Risk, because it has to do with how a project is structured: the parts that make it up.  Again, to take the positive angle – each point below is listed in its non-risk-adding, positive side:

• The team has worked together before. The team itself is a known quantity, with the individual talents and personalities of each team member known to the group.  This knowledge allows team members to guide the project to leverage the various strengths and avoid the weaknesses of the individuals on the team, working together to achieve the highest level of productivity.  Also hopefully people like and respect each other.
• The team knows the technology. It is a technology that the team has worked with before – they know how to work with it rather than wind up fighting it.  They are unlikely to be surprised by strange quirks of the tech stack that creep up at the most inopportune times.
• The team knows the client. The team has worked with this client before, and therefore knows the business problems that intersect their space.  This way they will be less likely to make poor assumptions about how things are supposed to work.  Functional specs are great, but they don’t take the place of actual domain knowledge on the part of the team.
• The team has a methodology for staying on track. Without getting into the comparative merits of individual development methodologies, the main point is that the team has a way to prioritize, track progress, and correct problems early on.  A team that can’t tell that it’s off-course can’t correct.
• The team can effectively communicate to management, and vice-versa. The cliche is that management understands different levels of priority for requirements, but sees all cost of implementation as equal.  Developers see different costs of implementation, but imagine all requirements to be equally important.  Neither is true, of course, and there needs to be an effective channel for technical and business stakeholders to properly communicate with each other.  A Tech Lead can be an effective method here.

Note that none of this discusses the individual experience or quality of the members of the team.  This is structural.  Process and management – all about getting people to speak with each other, and work well together.

Again 1 point for each “yes” answer for your team, just like in part one.

5 points: A. Your team knows each other, the client and the means to success like the back of their hand.  You’ve probably already made the mistakes you are going to make, so you can count on solid execution from people.

4 points: B. Still solid.  B is as good as a team can get with a new client (and at least in agency work new clients are frequent).  B is structurally still a good risk, but can occasionally throw up a surprise or two.

3 points: C. Medium risk.  At this level there is at least some degree of immaturity in the team – it may reflect new hires, or the creation of a new team where there wasn’t one before.  The team is still working out the kinks in their process.

2 points: D. This is an immature team.  There are enough unknowns going on at this level that the team can easily trip over itself, adding significant risk to a project.  Watch out.

1 and below: F. The team is not ready for this project.  The structural problems will create a significant negative impact on the project, probably leading to some level of failure.

To be concluded in part 3, where we’ll look at qualitative risk, and how to combine all the risk factors together into a single model.

Okay, no one buys that exactly, but there is a certain amount of seeming randomness in software or web development that makes developers and managers both very, very nervous when entering new endeavors.  Everyone knows from past experience that even with the best of intentions, things seem to occasionally go terribly wrong.

I’d like to suggest that things happen for a reason, however.  In the case of software development projects, there are specific patterns and phenomena that drive projects towards success or failure, and that learning to recognize those patterns can lead one to manage them much better.

In fact I would take that one step further, and say that these patterns can actually be modeled, and together form a bit of a magic 8-ball on what’s likely to happen with the project.

Clients

Many software development projects have clients involved, and for many of them the problems start right there.  Clients present an extremely powerful influence over projects, because if they are not happy, there is no business.  Clients do not always act in their own best interests however, or the best interests for the project.  Therein lies the rub.

I’ve identified a specific set of criteria that identifies good clients.  Clients with these attributes are good, those without them are bad.  Howe much risk a client brings to a project depends on how many items on this list they rate.

• Strong decision maker: Is there someone (much preferably an individual person) who can make decisions in the client organization?  Or is there a distributed set of stakeholders, all with different agendas, values and visions about what the product should be?
• Reasonably Insensitive to Price: Does the client appreciate the value of what they’re getting?  Or do they just want to slash, slash, slash prices and get the cheapest possible version of their site?  Do they argue over every last nickle and dime?
• Technology Agnostic: Tech matters.  Anyone who says differently is ignorant or lying.  The choices of hosting, frameworks, platforms, products etc all compound and add or remove risk from the project.  Good clients are ones that allow and encourage the best tools for the jobs to be used.  Bad clients come in the door with corporate technology mandates that have nothing to do with the specific interests of the project.
• On a Reasonable Deadline, but Flexible on Scope: Projects need schedules in order to happen, but the uncertainty baked into programming demands that something in the classic project management triangle flex, and the best thing to flex is scope.  When there is a clearly prioritized set of features on the table, the upcoming release may or may not have certain features, or they may be implemented at a number of different levels of complexity.  Hard features of limited business or user value should be jettisoned.
• Savvy: Good clients know their own business, the Internet and so forth.  Clients that require teaching on every last point that intersects the project slows things down, and opens the door for them to make bad decisions.  A client that is reasonably well educated on their space and how it relates to the project will be much less off than others.

Grading

For your client, award 1 point for each of the above criteria.

5 points: A. Run, don’t walk and get this client’s business.  They are extremely rare and can be the basis of an excellent partnership in which you can do great things.

4 points: B. A good, run of the mill client.  Will occasionally do something bothersome, but well within the parameters of something that can be managed for success.

3 points: C. Fair.  This is starting to get into the range where client risk can adversely affect the project.  If it is combined from risk from other factors, failure can occur.  Nonetheless it is possible to succeed with a C client.

2 points: D. Not good.  This is a high-risk client, and this kind of work should only be taken on if painstaking steps are taken to protect the team from bad influences and decisions made by the client.  A large amount of time will be spent on keeping this client under control, and preventing them from damaging the project.  It better be worth it.

1 and below: F. This client brings disaster with them.  It is almost impossible to succeed with this kind of client, and it usually isn’t worth it.  They will be a never-ending source of problems, and will compel bad decisions that will most likely derail the project.  Avoid.

Continued in part 2, when we’ll look at some of the other risk factors that bear on a project.

UPDATE: Okay, so I covered some of this before, but this is part of the model, so its sort of from the opposite tack.

A mathematician sits on a park bench next to a pretty girl.  He wants to move closer but is struck by the helplessness of the situation: he’ll move half the distance, then half of the remaining distance, then half of that, and so on and so on – but will never actually reach the girl because the remaining distance can always be halved.

Later an engineer finds themselves in the same position.  Like the mathematician, he knows its impossible to actually reach the girl, but he realizes he can get close enough for all practical purposes.

(Okay, it is, and has always been, a dumb joke.  But it’s about how engineers think, and that’s what I want to talk about.)

Engineers solve problems.  In order to solve problems, they need to identify them.  They need to get down in the detailed muck of whatever it is at hand so they can see how things work and find a solution.  This is called “getting your hands dirty”.

Most people are afraid of problems.  Problems make them feel like victims – helpless to control their fate.  They’re used to problems making life worse for them, without any available solutions.

Engineers, on the other hand, relish the thought of new problems.  After all, they’re used to having problems fall before their mental prowess – sliced into bits by Occam’s Razor.  The existence of problems is enjoyable to an engineer: it gives them something to do.  To most, however, the existence of problems is upsetting.

And that brings us to salespeople.  Salespeople identify problems that upset people, and they sell solutions to those problems.  (And since this is a web agency post, by “solutions” I mean “websites”).  In theory, people will feel better because their problems have now been solved. The salesperson also feels better because they now have money.

Now what salespeople don’t want to do is dwell on the problem..  They want to talk about the solution.  No problems should escape the solution.  The solution leads to the sale.  Focus on the solution.  Always Be Closing.  Dwelling on problems creates doubt in the mind of non-engineers, and doubt screws up closing the deal.

So, ultimately, the salesperson wants to play down any problems that are not solved by signing the deal, whereas the engineer wants to dwell on them in detail.  Disconnect.

Weird things can start to happen when you bring an engineer into a sales situation; something that is occasionally necessary.  In the web agency world, the art of deal making is that the sale is made long before it’s clear what is being sold.  For n dollars, something will be built that will do something on the lines of solving these business problems.  Sort of.  Give or take.

This is totally backwards to the engineer, who, in their problem solving mindset, wants to be given a problem to solve.  A set of metrics for success.  A clearly defined goal.  But that’s not the way agencies work.  First you pay, later you find out what you get.

In order to close the deal, the salesperson needs to keep out of the details, and basically imply that the website will do everything that the client wants, but without actually promising anything specific.  The danger to the salesperson is that the engineer, being detail conscious, will screw this up.

The danger for the engineer is that the salesperson might sell something that simply can’t be delivered for the specified budget.  Or perhaps commits the engineering team to a ridiculous technical constraint that adds so much risk to the project, or is so onerous, that what started as a simple project becomes a nightmare.

When an agency works well, there’s a kind of dance between sales and engineering, where sales gets deals done without being drawn down into the muck of engineering, but doesn’t create any deals too stupid for engineering to be able to deliver.

When sales and engineering don’t sufficiently trust or respect each other the dance breaks down.  Then sales closes deals that are impossible, and engineering bogs sales negotiations down in details that simply shouldn’t see the light of day until the ink dries on the contract.

The core of Disco is implemented in Erlang, the concurrent, fault-tolerant, multicore-ready distributed computing platform developed some time ago at Ericsson.  Erlang is brilliant, though kind of weird, and represents a big educational hurdle for the existing programming population.  It’s just too different from the existing major programming paradigms.

Disco takes a stab at solving that problem, by allowing programmers to write their jobs in Python.  The jobs are executed by the Erlang core, buying all that distributed, fault-tolerant goodness that Erlang provides, but keeping it safely sealed away from application developers who can work in the relatively friendlier world of Python.

Here (lifted directly from the Disco documentation) is a Disco job:

from disco.core import Disco, result_iterator
 
def fun_map(e, params):
    return [(w, 1) for w in e.split()]
 
def fun_reduce(iter, out, params):
    s = {}
    for w, f in iter:
        s[w] = s.get(w, 0) + int(f)
    for w, f in s.iteritems():
        out.add(w, f)
 
results = Disco("disco://localhost").new_job(
		name = "wordcount",
                input = ["http://discoproject.org/chekhov.txt"],
                map = fun_map,
		reduce = fun_reduce).wait()
 
for word, frequency in result_iterator(results):
	print word, frequency

So this code snip is about creating a word count of some text.  MapReduce always consists of two functions – the Map function, which is used to split up a big job into a bunch of smaller jobs, and the Reduce function which assembles it back together into a single result.  (This is the essence of MapReduce, and isn’t tied to a particular technology).

The code above has two fun_* functions.  “Fun” is a Erlang-ism that creates an anonymous function, not unlike a lambda in Python.  The functions themselves are passed into the Disco instance which then spits out the results, once all the reduce functions exit no doubt.

So in the above code example, it looks like each word gets its own job, zipping through the text and getting a frequency count.  The job split is initially established by fun_map.  Then fun_reduce runs, concurrently, once per unique word in the text and counts up the frequency of that word, adding its results to the “out” accumulator.  Disco ties it all together and returns it as the “results”.

Wait, this gets better.  Disco comes with tools that allow it to be deployed on Amazon’s EC2 computing cloud.  (Hm, Python.  Django-Disco anyone?) Imagine dynamic, linear capacity scaling, on rented compute cycles, with easily written Python jobs.  I think I might be salivating a bit.

I’m a huge fan of anything that can deliver concurrent programming power in a form that’s paletable to programmers that haven’t grown up with it.  I’m going to eagerly watch the Disco project to see how it does.

That’s why I’m absolutely confounded by people who, when being presented with the opportunity to learn something new on their employer’s dime, refuse or drag their feet.  These people have someone who is willing to pay them to become even more marketable than they are now.  They can earn while they learn.  And yet, their answer is “no thanks”.

What drives this kind of thinking?  Why would someone decline to learn something new?  I can think of a couple of things:

First, I think that people legitimately realize that learning new things can potentially change your worldview, making you re-evaluate the truth of things that you supposedly knew before.  However, this becomes the “path to the dark side” when people reject the idea of learning new things because they’d rather maintain their existing worldview rather than having a greater comprehension of the truth.  It’s willful ignorance.  To avoid learning things for this reason is to choose to be stupid.

Secondly, (and I’m a bit more sympathetic to this reason) some people have been taught that a particular technology as a solution to everything.  Java (where I spent a good decade of my time) has been sold as the development environment for everything from an embedded, to a desktop apps, to web apps, to mainframe computing.  The message was that Java is the tool of choice for everything.  So why would one learn anything else?  What’s the point?  (Okay, so this actually sounds a bit like laziness to me, but such is the human condition – I accept that.)

Finally, a somewhat plausible reason I’ve heard is that people only want to learn things that have a direct impact on their capacity to make money.  Java developers are statistically paid the most of any developers – so again – why learn something else?  What is sad about that is that the best of the development community (and the source of much technology) comes from people who actually love technology.  People who are only in it for the money tend to also be mediocre programmers.

So here’s my plea to those who would choose not to learn new technology (or new things in general), whether out of fear, laziness or avarice: your brain is dying.  Save it.

Learning is a habit that keeps you sharp.  The more you learn, the more capable you are of learning.  This is a virtuous cycle that can get you 1) a robust and well-informed (albeit still disrupt-able) worldview, 2) a deep toolkit of knowledge to take on different problem domains and 3) awesome market value, allowing you to rake in the cash.  In other words – all of the stuff you care about can be accomplished to a much higher degree by learning new things.  There’s work involved, sure, but if you stay ignorant you’ll only be working hard anyway: putting out fires in your career because you don’t have the knowledge to take control of the situation.

Taking every chance I’ve had to learn new things has been the engine that’s driven my career.  So, please, if someone offers you a chance to learn something new, on their dime, take it.  On both a personal and professional level, its keeping you alive.

The idea of maintaining a single technology stack in a development shop is quite appealing.  If everyone knows the same programming language (the thinking goes), the same web framework, the same RDBMS and so forth, then life would be better.  Business really likes this kind of thinking.  It promises efficiencies of scale, reduced training costs, and easily replaceable employees.  This kind of “one true technology” approach is the foundational philosophy of corporate darlings Java and .NET.    Once you have the organization standardized on the tech, it is promised, then every part will play nicely with every other part, critical dependencies (such as employees) will be reduced, and the whole “what are we doing about technology” question just goes away.

Now, of course I support operational efficiencies, reducing training costs (well, to an extent: I view training as an investment), and so forth, but I think that the idea of “one true stack”, especially in service-oriented tech shops like web agencies, is a pipe dream.  The advantages of standardizing an entire company on a single tech stack are outweighed by the disadvantages.

Before I get into that, however, I want to introduce the idea of a “jealous” technology.  A jealous technology tries to take over as many aspects of the stack as possible; mandating a specific component at each level.  Some technologies are more “jealous” than others.  Microsoft is notoriously jealous:  the Microsoft web framework, ASP.NET, really wants to be deployed on the Microsoft web server: IIS, on the Microsoft operating system: Windows, backed by the Microsoft RDBMS: SQL Server.  Other combinations (yes, I know about Mono) simply don’t work as well.  And, on the other hand, Windows is something of a second-class deployment platform for most other web technologies (ever try to deploy Rails on Windows…with IIS?  Yuck…)

Speaking of Ruby on Rails, they’ve become a bit of a jealous technology themselves.  Rails projects now prefer a particular JavaScript framework (Prototype), version control system (Git), editor (TextMate), build system (Capistrano) and development operating system (OS X).  (As well as, obviously, programming language: Ruby). While it’s certainly not as bad as the Microsoft scenario,  I do have to ask the question as to why the web framework has any influence over what version control system I use.  Do you have to use all the technologies that I’ve listed above with Rails?  No, of course not.  But they are preferred.  Things go better with Coke Capistrano.

The point is that some technologies are better suited towards certain problems than others, and a blanket organizational standardization on a single tech stack leads to a tremendous amount of wasted energy spent on driving square pegs into round holes.  Ruby is not a great foundation for an enterprise-class messaging framework.  Java is not a good choice for a lightweight CMS.  And anything other than .NET is a bad choice for an app destined to be maintained by a bunch of MSDN types.

In fact, I’d argue that the cost of operation efficiencies lost by maintaining multiple tech stacks is exceeded by the cost of jamming technologies into inappropriate use cases.  So, as messy as it sounds, any technical organization that is project or service based (as opposed to being exclusively product based) should maintain a strategic “quiver” of different technologies, so it can deploy the best choice when faced with a business problem.  Otherwise a fragile monoculture is created, wasting a tremendous amount of developer effort, and adding artificial risk and expense to projects.

The catch is that we don’t really know how to program for this.

To take advantage of massively multicore architectures, we really need to take our computing problems and decompose them into smaller bits that can be farmed out to various worker processes.  This idea is what Google’s MapReduce is all about, and is an approach that has worked well with certain categories of problems, such as bioinformatic data analysis, or 3D rendering.

Programming Crisis

I am seeing a pending programmer educational crisis coming.  Most programmers just are not trained to think about this kind of decomposition.  They’ve learned functional and object-oriented languages, and tend to think in a linear fashion.  Furthermore, the nature of the programming languages and platforms they’ve  worked on has ingrained the concept into them that multi-threaded programming is extremely tricky.  And with mainstream platforms, it is.

A lot of the existing difficulty with well-known languages, such as Java, is managing shared state, or memory between different threads.  Java has various synchronization mechanisms, but using them in heavily multithreaded environments can be challenging.  I’ve seen unit tests randomly pass or fail, swinging on race conditions – which thread ended first – that determine the success or failure of the test.

In the web world we’ve become rather fond of very high level languages such as Python and Ruby.  However the runtime story gets even worse – most scripting language runtimes can’t acknowledge more than one processor at a time – meaning that the only way to parallelize work is to instantiate multiple instances of the runtime.  Besides this being a hack, it makes communication between processes awkward (usually relying on expensive data serialization), and creates the kind of issues that we’ve been trying to get away from with high-level languages: the developer has to deal with boilerplate computer science problems, rather than being allowed to exclusively concentrate on their business problems.

Concurrent Languages

One answer seems to be tech stacks that are designed for concurrency from the ground up.  Enter Erlang, a language developed for telephony applications by Ericsson.  Erlang handles concurrency very, very well.  For example, it circumvents the shared state problem by not having any shared state.  However, from the perspective of someone with a Java or Python background (e.g. me) it’s frickin’ weird.  Some serious adjustments in the way problems are approached have to be made.

Here’s a real simple code snip, that generates the squares of a list of numbers.  First a naive implementation in Python:

def squares(num_list):
    square_list = []
    for n in num_list:
        square_list.append(n * n)
    return square_list

A more compact version in Python, using a list comprehension:

def squares(num_list):
    return [n*n for n in num_list]

Here’s the same code in Erlang:

square([H|T]) -> H*H | square(T);
square([]) -> [].

Assuming you don’t know Erlang, can you even tell what’s going on? You’re looking at a completely different paradigm. The left side of the functions (to the left of the “->” marker) relies on pattern matching (sort of like regular expressions).  To the right side, the function implementation is tail recursion at work.  Needless to say, this is a bit of a departure for many developers.  I shudder to think of trying to take a department of developers who cut their teeth on Java and PHP, and train them up in Erlang.

This is why I’m so interested in projects like Reia.  In an attempt to fix the “impedance mismatch” between the coming massively multicore future, and today’s programming skills, their goal is to make a high-level Python/Ruby like language that compiles onto bytecode that will run on the Erlang VM.  It’s in a very early stage, and I’m not sure if it will ever gain critical mass, but this is a problem that needs solving, and I’m intererested in any attempts.