The Software Management Blog

Groovy and Grails Training in VA

You may have heard that Groovy was the fastest growing language of all times, attracting fans from around the world. OK, I made up the first part, but its true that bunch of cool people are coding lots of good stuff in Groovy these days. The language and process gurus Scott Davis and Andy Glover are teaching a 2-day Groovy and Grails course in Reston, VA. I wish I could attend as I am sure it will be informative and fun.

Agile 103: Continuous Integration and Agile Development

Compared to other disciplines, software engineering is still in its infancy and as such, it is undergoing significant transformations. This is a challenge because of the relentless innovation and ever changing standards; it is an opportunity because the low barriers of entry make it possible for virtually anyone with a computer to actively participate in the revolution and change course in significant ways.

Let’s take development methodologies for example. In the beginning of the century few people had heard of Agile development. Today, it is growing in popularity and adoption numbers suggest its value and effectiveness.

The Agile movement started largely out of necessity by looking at ways to increasing efficiency by reexamining the way people communicate and create software. The Agile model was explained as a collection of simple principles and practices. One of these practices is Continuous Integration (CI).

Continuous Integration (CI) is the practice of executing series of build steps in a scheduled or automated fashion. The build scripts may be run by a computer or a human; they can be automated or manual. The important thing is to run them often and provide the team with valuable feedback as soon as they need it.

CI can be defined as an umbrella practice enclosing sub-practices such as Continuous Builds, Continuous Database Integration, Continuous Unit Testing, Continuous Deployment, Continuous Functional Testing, Continuous Notifications, and Continuous Reporting. All of these are designed to shorten the time between the problem discovery and the solution. Agile teams can rely not only on the time-saving nature of automated builds, but also on the timely and accurate feedback. Depending on the CI configuration, when one of the steps fails the CI server may choose to continue or suspend further operation until the failure is taken care of.

The typical CI system starts with the compilation and packaging of the application. Any problems at this stage usually cause the CI cycle to fail and the system to notify the submitter or the entire team of the failure. Once the problems are fixed the CI server recompiles the source code and moves on to executing any Data Definition Language (DDL) and/or Data Manipulation Language (DML) scripts as part of the database integration step. This ensures the database gets synced with the source code and that any unit and functional tests that rely on the database changes execute properly. Depending on the server configuration unit test failures may or may not fail the CI cycle. If the application requires packaging and deployment, the CI server packages and deploys the archive into a container. After functional tests pass the CI server labels the source code with an automatically generated label. Finally, it generates a report of the integration cycle and notifies a list of recipients.

CI is an important practice for the Agile team. The CI system can provide feedback to all parties immediately after a problem is discovered. It can also generate valuable deployment artifacts and reports for various roles within the Agile team. Finally, the CI system brings teams closer to having the ability to plan faster and more frequent releases at the end of each iteration.

Way to go!

I’d like to take a moment to congratulate my friends Paul Duvall, Steve Matyas and Andrew Glover on their winning of the 2008 Jolt Awards in the technical books category for their exceptional work “Continuous Integration: Improving Software Quality and Reducing Risk“. It’s a well-deserved award for an exemplary book from which I learned a lot and one that I recommend to all of my clients. Good job guys, way to go!

PHP 101: Setting-up an Environment for Development and Debugging

After years of developing “enterprise” [read:big and bulky] software you are tired from the whole shebang J2EE/.NET and you are now looking for easy lightweight web development. PHP can be one of your choices.

How do you start? Easy, just follow these steps to quickly setup a development environment with all servers, IDE and a debugger. In a few quick steps you will be able to setup:

• XAMPP which comes with:
1. Apache
2. PHP 5.2.x
3. MySQL 5.0.45
4. Mercury Server (SMTP, FTP, etc.)
• Zend Studio for Eclipse

Let’s start:

1. Download XAMPP 1.6.5 for Windows from http://www.apachefriends.org/en/xampp-windows.html#641
2. Follow the intuitive setup wizard to install
3. Download Zend Studio for Eclipse from http://www.zend.com/en/products/studio/
4. Install Zend Studio
5. To enable the Zend Debugger first navigate to XAMPP_HOME\php\zendOptimizer\lib and create a new folder Debugger. Create a new folder php-5.2.x under Debugger.
6. Copy ZendStudio_HOME\plugins\ org.zend.php.debug.debugger.win32.x86_5.2.12.v20071210\ resources\php5\ZendDebugger.dll to XAMPP_HOME\php\zendOptimizer\lib\Debugger\php-5.2.x
7. Open XAMPP_HOME/apache/bin/php.ini and add disable the Zend Optimizer while enabling the Zend Debugger:
   
[Zend]
zend_extension_ts = "XAMPP_HOME\php\zendOptimizer\lib\ZendExtensionManager.dll"
;zend_extension_manager.optimizer_ts = "XAMPP_HOME\php\zendOptimizer\lib\Optimizer"
;zend_optimizer.enable_loader = 1
;zend_optimizer.optimization_level=15
;zend_optimizer.license_path = "XAMPP_HOME\php\zendOptimizer\lib"
zend_extension_manager.debug_server_ts = "XAMPP_HOME\php\zendOptimizer\lib\Debugger"
zend_debugger.allow_hosts = 127.0.0.1
zend_debugger.expose_remotely = always
8. To verify the debugger, navigate to http://localhost/xampp and click the phpinfo() link. You should see the Zend Debugger information on the page such as
This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
with Zend Extension Manager v1.2.0, Copyright (c) 2003-2007, by Zend Technologies
with Zend Debugger v5.2.12, Copyright (c) 1999-2007, by Zend Technologies

9. In Zend Studio, create a PHP project or use the sample project to start debugging
10. Keep in mind that Zend Studio has two debugging modes: local and remote. In this example both are enabled. If you selected to install the Zend Firefox extension you can launch a debug session right from Firefox. Nifty.

Quick Tutorial: Installing Subversion on Linux

This tutorial explains how users with no root access can install Subversion onto a shared Linux server.

System specs:

• Linux 2.6.22 or higher
• Subversion 1.4.5
• Apache Portable Runtime (APR)
  • APR 0.9.17 - http://government-grants.org/mirrors/apache.org/apr/apr-0.9.17.tar.gz
  • APR-util 0.9.15 - http://government-grants.org/mirrors/apache.org/apr/apr-util-0.9.15.tar.gz
• neon - an HTTP and WebDAV client library - http://www.webdav.org/neon/neon-0.25.5.tar.gz

Step by step installation guide:

1. Start by checking the Linux version of the host to which you’re going to install Subversion

   # uname -a
   Linux myhost.com 2.6.22-9_1.BHsmp #1 SMP Fri Sep 28
   23:36:16 MDT 2007 x86_64 x86_64 x86_64 GNU/Linux
   

2. Install neon

   # wget http://www.webdav.org/neon/neon-0.25.5.tar.gz
   # tar -xzf neon-0.25.5.tar.gz
   # cd neon-0.25.5
   # ./configure --enable-shared --prefix=$HOME
   # make
   # make install
   

3. Install APR

   # wget http://government-grants.org/mirrors/apache.org/apr/apr-0.9.17.tar.gz
   # tar -xzf apr-0.9.17.tar.gz
   # cd apr-0.9.17
   #  ./configure --prefix=$HOME
   # make
   # make install
   

4. Install APR-util

   # wget http://government-grants.org/mirrors/apache.org/apr/apr-util-0.9.15.tar.gz
   # tar -xzf apr-util-0.9.15.tar.gz
   # cd apr-util-0.9.15
   # ./configure --prefix=$HOME --with-apr=$HOME
   # make
   # make install
   

5. Install Subversion

   # wget http://subversion.tigris.org/downloads/subversion-1.4.5.tar.gz
   # tar -xzf subversion-1.4.5.tar.gz
   # cd subversion-1.4.5
   #./configure --prefix=$HOME --without-berkeley-db --with-zlib --with-ssl
   # make
   # make install
   

6. Verify installation

   username @jacoozi.com [~]# svn --version
   svn, version 1.4.5 (r25188) compiled Jan 18 2008, 12:24:06
   Copyright (C) 2000-2006 CollabNet.
   Subversion is open source software, see http://subversion.tigris.org/
   This product includes software developed by CollabNet (http://www.Collab.Net/).
   The following repository access (RA) modules are available:
   * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
     - handles 'http' scheme
   * ra_svn : Module for accessing a repository using the svn network protocol.
     - handles 'svn' scheme
   * ra_local : Module for accessing a repository on local disk.
     - handles 'file' scheme
   

Agile 102: Continuous Integration Explained

One of the points in What is Continuous Integration (CI)? was the importance of having reliable automated build scripts and executing them often. CI can be seen as a natural extension of the Automated Builds practice since in most cases it relies on the presence of a set of ABS. A well-designed CI system encapsulates other sub-practices such as Continuous Builds, Continuous Database Integration, Continuous Unit Testing, Continuous Deployment, Continuous Functional Testing, Continuous Notifications, and Continuous Reporting. All of these are designed to shorten the time between the problem discovery and the problem solution. Agile teams can rely not only on the time-saving nature of automated asynchronous builds, but also on the timely and accurate feedback as a result of this type of CI.

Continuous Integration (CI) is an umbrella practice enclosing a sequence of logically related steps. Depending on the CI configuration, when one of the steps fails the CI server may choose to continue with the remaining steps or suspend any further operation until the failing step is fixed.

The typical CI system starts with the compilation and packaging of the application. Any problems at this stage will usually fail the CI cycle and the system will notify the submitter of the entire team of the failure. Once the problems are fixed the CI server will recompile and mode on to executing DDL and/or DML scripts as part of the database integration. This will ensure that the database will be up to date for the unit and functional tests that will follow. Assuming there are automated unit tests, the system will make sure to create test data and execute the unit tests in the suite. Depending on the server configuration unit test failures may or may not fail the CI cycle. In the next two steps the CI server will package and deploy the application. After all functional tests successfully pass, the CI server will label the code with an automatically generated label, it will notify a list of recipients and will generate a report of the integration cycle.

An Agile team can benefit from such a system in multiple ways. For one, the system can provide an immediate feedback to all parties shortly after a problem is discovered. Secondly, it can create valuable deployment artifacts and reports targeting different audiences within the Agile team.

One variation of the above diagram is if the CI server is configured to run code inspection tools as part of the CI cycle. Static analysis tools run from the CI server can generate reports on code metrics such as lines of code, test coverage, cyclomatic complexity, package coupling and dependencies, and can provide important data points for the entire team as to where the system stands in terms of code quality and architectural integrity.

A second variation of the diagram can happen when successful completion of the functional tests triggers a second deployment to another machine. Since the CI server knows it is a good build, this time it deploys to a user acceptance machine, targeting the customer directly. The advantage of redundant deploys is that semi-independent teams such the User Experience (UE) have a machine that always runs the latest good build.

Reference: This blog entry is derived from author’s own real-world experiences building CI systems and industry best practices. One of the leading sources for CI best practices is Addison-Wesley’s recently published Continuous Integration, a Martin Fowler Signature Series title, authored by Paul M. Duvall with Steve Matyas and Andy Glover.

ROI of Agile Development

Scott Sehlhorst at Tyner Blain has added an excellent article on ROI in Agile. Take a read!

Agile 101: What is Continuous Integration?

What is Continuous Integration (CI)?

Continuous Integration (CI) is the practice of periodically executing Automated Build Scripts (ABS), preferably in short intervals. There are two flavors of this practice. The first flavor, mostly applied by early XP practitioners states that the system is built by one developer at any given time. The teams have adopted various physical tokens signifying the builders’ "right of build". In this flavor of CI Automated Build Scripts are still required, albeit the kicking off the build happens manually. Since this version relies heavily on the entire team to be colocated, it does not scale well in distributed Agile projects.

The second flavor of CI relies on a computer background process to poll the code repository in short intervals and automatically trigger the build process if it detects any changes since the last build. In this version of CI, the developer is only responsible to build locally and check-in the code. This model is better suited for highly scalable Agile teams because of its reliance on automated asynchronous processes to trigger the build. This gives distributed teams an early feedback on the status of the system and therefore it enables dislocated team members and managers to have the same quick feedback as the team itself.

Distributed Agile Development

Team Structures in Onshore-Offshore Distributed Agile Development Teams (OODT)

One of the biggest challenges of offshore development is finding a reliable partner. Unless the US company owns the operations overseas and regardless of the engagement model, special care should be taken to start with a competent and trustworthy partner. If the offshore partner or its employees have prior US work history and some Agile development experience, this will increase the chances of success for the OODT Agile initiative.
Flexibility is another important attribute and despite flashy marketing materials, not all of the offshore companies have developed habits of business savvy and flexibility.
Try to identify an offshore partner willing to work flexible hours to accommodate for the time zone differences. Activities such as testing and deployment can be seen as separate and therefore can be run on a separate schedule, but that does not scale well within the Agile team, especially if it practices the Whole Team. Therefore, effort should be taken to make the whole offshore team available to the onshore team for a portion of the day.

Now, let’s take a look at some possible variations of the offshoring model. There are probably as many models as thinking heads, however it is possible to classify them as partial and complete offshoring. Before looking into the details of each model, let’s take the composition of the typical colocated Agile team as a baseline and compare it to three possible models for offshoring.

Baseline: Colocated Onshore Agile Team

Let’s start with a typical colocated team where all team members are located in a US-based facility. The diagram below is presented as a baseline for comparison with the three possible offshoring models.

Model 1: Partial Offshoring

This is the first model of offshoring where parts of the functional team are shipped overseas. In this model the design team is located onshore and it sends the requirements overseas. The offshore team responds with back fully-tested and working code.The onshore team then performs the user acceptance tests and validates the requirements and the defects.

Model 2: Complete Offshoring

The second model tries to provide as much colocation as possible by shipping the entire production team overseas. The only person left onshore is the customer. The customer sends features and the offshore team responds with fully tested working code. This scenario is arguably the most cost-effective among all.

Model 3: Mirrored Offshore Team

The third and most effective model for companies transitioning to Agile is the mirrored-design model. This model can be less cost-effective in the beginning, than Model 2, however it the initial investment pays itself out with less defects and shorter development cycles. In addition, the mirrored teams lay the groundwork for a smooth transition by gradually shifting responsibilities to the offshore team.

Agile 101: Automated Build Scripts (ABS) defined

What is ABS?

Automated Build Scripts (ABS) provide an ability to build software in an unattended and machine-independent environment. ABS are generally written and maintained by a special group of developers called configuration managers, or software integrators or just application builders. Some of the tools used to write ABS are Ant, NAnt, Maven, MSBuild, Visual Build Pro, and make.

Automated builds are one of the most important practices for any Agile team. They help developers see the effects of any change immediately and make it possible to deliver working software at the end of each iteration. The need for ABS is even more pronounced in distributed Agile teams because multiple people build simultaneously and in the absence of a common team room it is paramount that every change is properly integrated into the build. This is a fundamental attribute of a well designed automated build system since it helps keep the repository coherent and the code base buildable.

Other important features of good ABS are:

• Allow a system to be built on any machine while requiring minimal (properties) or no changes to the base scripts - in most Agile teams it is more than likely that teams will have different machine settings such as OS, language, IDE. The ABS should be machine- and environment-independent.
• ABS should be easy to plug-in to a Continuous Integration (CI) sever - this is especially important for distributed and offshore Agile teams as the time zone differences make it necessary to build continuously and notify the teams of any problems early. ABS written in Ant or Maven for instance, allow for automated and unattended execution, making them a perfect choice for CI.
• Ability to execute ABS with a simple shell command - both the architect and the developers should be able to trigger a full build with a simple shell command. The best ABS are designed to be simple and guiding the user through the steps so that even relatively non-technical team members such as business analysts and project managers are able to stand-up a system with minimal help from the rest of the team.
• Ability to define and manage project dependencies. These are both compile-time dependencies such as libraries as well-as deploy-time dependencies such database scripts and XML configuration files. In highly collaborative Agile projects the DB servers and XML configurations are likely to change often, so a good ABS should take care of the dependencies in a transparent manner.