Adventures in home working

I thought I would pick two previous projects and look at their potential as Open Source projects.  The second I picked is a bit whacky.  It was written almost entirely in DCL and provided an automated help desk job logging, analysis and reporting tool and knowledge base.  To write this in DCL is a testament to the flexibility and power of the VMS scripting environment and indexed files and the creativity of one of its developers, (not me).  But the concept was mine and its ‘conceptual integrity’ was maintained for many years.  What value could this have had in an Open Source context:

1. The application itself had little value, although it was trivial to deploy and could have proved very popular for other small support teams.
2. The tools developed to manage a complex system constructed from DCL would have been very valuable
3. The library of DCL routines would have been very useful to the community
4. The concepts used in the system which allowed 4GL like application development may have inspired other developers to rapid application prototyping of similar applications.

This system lived for well over a decade and died because the team that used it were disbanded,  had that not happened it may have continued to spiral out of control through entropy like neglect which Open Source enthusiasm and stewardship would/might have prevented.

As I gradually migrated towards infrastructure and away from line of business applications, the reality of having to deliver applications to Windows and manage them on Windows began to dawn on me.  To a developer used to centralised computing, with remote access through X Windows or terminal clients this was a considerable shock.  However my first real Windows development project showed one of my most valuable character traits, I don’t give up easily!  Without going into the gory details, here are some of the attributes of that first application, a system for capturing, storing, accessing and viewing large image collections:

1. Very easy deployment
2. Self maintaining code, i.e. a minimal system start-up application compared what was installed with what should be installed according to a central manifest, and updated itself accordingly.
3. A very flexible storage model built on the concept of logical storage units, (a bit like VMS logical names on steroids), which handled the fact that images could be on removable CD’s, local disks, media libraries, networked disks etc and in different combinations.
4. Data driven.  The whole system was configured through simple text files and meta data definitions that defined the actual data structures in the SQL database.
5. Globally unique naming of files, to avoid cache conflicts between image collections from different sources accessed at the same client

The result was an incredibly cheap system to purchase and deploy, that could run on a laptop or on 1000 networked desktops with the differences all defined in its data definitions.  Data collections could be distributed on CD, with updates on floppy disk, or networked but transparently cached for fast WAN access.

My team, (a very small one), built this system making extensive use of freely available software libraries, (for Visual Basic), in a very short period of time, and by having a small team with a leader with a clear vision we kept the concept intact from design through to initial deployment.  Although like all systems entropy resulted in some degradation of that concept over time, resolved only when a friend of mine reinvented the system as a web application with similar clarity of vision.

In this article I use Open Source in its broadest sense, (i.e. not consistent with the specific licensing defined by opensource.org), but meaning my experience in using other peoples source or developing source for others to use.   In particular I wanted to give examples of where the concept of Open Source had it been so visible then, would have suited some of my projects.

My computing true home has, and always will, be VMS.  As a VMS systems integrator I learned rapidly to admire the power, elegance and consistency of the system, its ‘conceptual integrity’ if you will.   It’s this history of VMS that made it so difficult to admire Unix, which by contrast has always seemed lacking in that same ‘conceptual integrity’, to stretch a point, always seeming to have been assembled rather than architected.

It’s also that history of VMS that first attracted me to Windows NT, (the development of VMS was led by a team from Digital who worked extensively on VMS).  Having been repelled by DOS and Windows, I was excited by the prospect of NT but ended up being left gaping at the lost opportunity, (still not realised today).  I was truly shocked to read the early history of NT and discover the neglect of that sacred VMS like architectural vision for the platform as a whole.

This neglect seems to have been caused by two key issues

• the focus that Dave Cutler and his team had on the kernel at the expense of everything else
• the need to achieve compatibility with the DOS/Windows/OS2 legacy. 

Had Bill Gates recruited a different, or at least more balanced, part of the Digital VMS team the world might be a very different place.  Of course, pre-Linux,  I realise that the Unix community saw a similar lost opportunity seeing the progressive dominance of the inferior NT and MAC/OS, (now corrected), whilst the superior Unix was increasingly marginalised.

It’s through Digital that I first experienced the web and deployed by first Intranet and obtained my first CD full of Open Source, (actually GNU), software tools and utilities, (for VMS).

This is the first real post in the Open Source category of my blog, but one of many in the blog as a whole, and one of many to come on Open Source.  If you have read my blog from the beginning you will know that much of my recent experience is with Enterprise Infrastructures, and that has to a large extent involved software from IBM/LOTUS and Microsoft.  Integration with Unix systems being through well defined and mature interfaces like NFS, X-Windows and DNS.  This means that I have a lot of familiarity with Microsoft and I am open about the fact that there are things to admire about Microsoft, (and many things to not admire of course), however I have a long held low tolerance for Zealotry which I have talked about previously, but want to expand on here.

When I say I dislike Zealotry, I am not talking about passion, I am talking about taking a stance for or against something that can not be defended by rational argument.  This causes me a problem, because as an enterprise architect, I spend my life having to defend the decisions I take with rational argument, and for me, ‘I hate Microsoft’ or ‘I hate Unix’, just does not pass the test.  Second as an architect I have to recognise real world constraints such as legacy applications, investment prioritisation, application compatibility, cultural aversion to change or risk etc in my solutions.  These two factors have shaped most of my thinking around the Linux/Windows and previously Unix/Windows debate.

However that is changing, and this blog will chart that change.  However before I dive into that I want to provide two of the main reasons why Open Source has become important to me personally and also professionally.

Reason 1: Credibility

I have noticed an increasing number of well respected Microsoft employees or expert system integrators either migrating to Linux or becoming expert in both Windows and Linux.  These individuals seem to have maintained a rational view of the world and are able to develop solutions recognising the strengths and weaknesses of both platforms.  It’s also interesting to see this happening in other areas, a good example being the .NET/Mono community who talk of the elegance of the .NET framework, whilst at the same time criticising Microsoft for lacking the vision to provide cross platform support.   I want to ensure that I am more than credible as an architect, open to, and conversant with the benefits of Open and Closed Source alternatives and their respective merits.

Reason 2: Concern

I have become increasingly concerned about the prospect of the worlds infrastructure being run by the proprietary technology of a single supplier.  I increasingly feel that the world’s IT infrastructure needs to be owned by the world, or at least the standards should be owned by the world.  That’s not to say that I am confident that the mechanisms are in place for this to happen naturally with Open Source as it is delivered today, but more of that in later articles.

Professionally I find myself increasingly being asked to provide advice on Open Source alternatives and to develop solutions based on them.

I described my Office in a previous post.  In this post I thought I would provide a bit on insight into my home network.  The following diagram should give you the basic idea.

1. The hub of the network is a little 4 port 100MB switch.  All three servers and one laptop are plugged into this
2. There are two dedicated servers on the network
   1. Server 1 is a dedicated application server.  It only runs Windows 2003 Server and GSX Server 3.1.  All application servers and some test desktops run on top of GSX server.
   2. Server 2 is a dedicated management server, and Active Directory domain controller.  This server runs the MOM Express 2005, and acts as a backup of my main file server, using Windows 2003 Volume Shadow Copy.
3. My desktop PC also happens to run Windows 2003 server, as this allows me the flexibility to access it via Windows Terminal Services from anyhwere in the house, mirror my data files, and also runs as AD domain controller for resiliance.  It also runs VMWare Workstaton.
4. When I need access to my company network, I run up a corporate standard PC in a VM on my Desktop Server and VPN from there, (VMware allows me to share files between my Corporate network and my PC network, with full network isolation).
5. The Pinter/Scanner is plugged into the management Server and the Desktop application server, one connection via USB and the othere via parallel.
6. For flexible use around the house and offsite I also have a laptop which runs Windows XP Professional, but can access Linux from the VMWare application server.  The laptop and my PDA both connect via WIFI.
7. My eldest daughters both have PC’s.  One laptop and one desktop both connected via WIFI.
8. All machines are part of the home AD Domain and use roaming profiles.
9. All data access except from my desktop server is via DFS
10. All print access is via my desktop server
11. Windows Terminal services is used extensively by everyone in the family
12. My desktop has three screens all driven from my desktop server.  One display via analogue, one via DVI and one via a virtual display adaptor from Maxivista.
13. I have a KVM switch that allows my Keyboard, Mouse and Secondary display to be connected to either my desktop server or my application server for console access, although its more convenient most of the time just to leave a Windows Terminal Server session open.

In a future post I will explain a bit about my home lab, which is all virtual for maximum flexibility and runs a mix of Windows and Linix servers and clients.  I will also expand a bit on my desktop application environment and security.

Microsoft research have come up with some really cool tools for capturing and manipulating whiteboard contents captured using low quality web cams.  My whiteboard is right behind me, (so my web cam points right at it, so it would work just great, but the downloads are MS only.  The best trick is it removes the person writing on the board from the image.  Here are some of the key points:

Other systems use expensive cameras or dedicated electronic whiteboards. The Live Whiteboard system, developed at Microsoft Research by Zhengyou Zhang and Li-wei He, uses whatever whiteboard you already have. It only needs an inexpensive Web cam and some clever software.

Live Whiteboard doesn’t just deliver a video stream of the whiteboard. The software takes out all the shadows and uneven surfaces that come through on a Web cam, and turns the whiteboard into an image that allows viewers to see the whiteboard notes. Through a series of image processing procedures, the originally captured image is first transformed into a rectangular bitmap to correct perspective distortion, and then color-enhanced to increase contrast, saturation, and to provide a clean uniform white background.

In addition, if the remote viewer wants to focus his attention only on the content, the system can take out the image of the person who is writing on the board. The remote viewer sees only the new content magically appearing, he never sees the person who is writing the content. This saves even more bandwidth.

The full news article can be found here, and the web site for the developer with more info and the research reports can be found here.

I have thought for a long time that Microsoft don’t make much use of their own software to build pre-integrated solutions for their customers, (unlike Oracle for example).  They seem to have caught onto the idea at last, (not from listening to me though ).  Anyway a few months ago they started to talk about solution accelerators, which are solutions built from sets of MS products with associated processes, procedures and best practices as well as custom systems integration.  These solve particular business problems, like for example, the process of hiring new employees.  There is also evidence that Microsoft is doing the same at the infrastructure level where the range of tools available to them is even richer, SQL Server, BizTalk, SharePoint etc.  This is a good example:

Microsoft also plans to make available to enterprise partners a “zero touch provisioning” accelerator that will enable end users to self-service tasks such as requesting the installation of an application or resetting a network password.

“We built in a rules-based engine based on BizTalk that can automate requests, get approved by a manager, and install a new application,” Hassall said. “And the opportunity is not just for desktop deployment but add-ons for server infrastructure using SMS and Active Directory and BizTalk in providing an infrastructure for installation and provisioning services.”

The company also plans to make available to enterprise partners a “zero touch provisioning” accelerator that will enable end users to self-service tasks such as requesting the installation of an application or resetting a network password.

“We built in a rules-based engine based on BizTalk that can automate requests, get approved by a manager, and install a new application,” Hassall said. “And the opportunity is not just for desktop deployment but add-ons for server infrastructure using SMS and Active Directory and BizTalk in providing an infrastructure for installation and provisioning services.”

Microsoft’s latest marketting phrase – Integrated innovation is starting to mean something.  If you want to see more on this topic then enter “integrated innovation”, into the search box on the left, include the quotes!

A new version of open office is available.  The main improvements are:

Enhancements to the open-source productivity suite include support for PDF and XHTML exports and improved compatibility with Microsoft Office, according to the OpenOffice Web site. The new release, for example, will support forms conversion within Word documents and import text document layouts with more fidelity. OpenOffice 1.1 also boasts enhanced support for mobile device formats such as Palm’s AportisDoc, Pocket Word and Pocket Excel.

IBM has ideas of its own, taking a thinner approach with its WorkPlace products

A wild card in the Office wars is IBM, which plans to offer server-based word processing, spreadsheet and presentation functionality to buyers of its WebSphere portal. At the very least, that could allow large customers to negotiate better Microsoft Office pricing/licensing, observers said. (See IBM Plans Sneak Attack On Microsoft Office.)

The MS Office team are majoring on quality for their next release, does this imply major changes, requiring major testing, or just good practice?

Software development, especially for a product as feature-rich as Office, is a repetitive process comprising what can seem to be endless feedback loops and rework.

“We’re trying to reduce the iteration of that cycle because it’s extremely costly,” said Steven Sinofsky, senior vice president of Microsoft’s Information Worker Product Group. “We want to use our development resources more effectively, yielding higher-quality code and not iterating what customers never see,” he said.

The Office 12 team will rely on new tools, including Buddy Web, a system developers can use to privately share releases, according to the memo, from Eric Fox, Office development manager at Microsoft. Buddy Web had previously been used by the Outlook team.

In addition, the Office group will have access to Big Button, a system that gives developers easy access to the appropriate set of tests for their code.

Office 12, will not reply on Longhorn, not really a suprise, but its in print.

Microsoft knows it would be folly to leave the hundreds of millions of Windows XP and 2000 users out in the cold and force an upgrade to the shiny, new and radically different next version of Windows, code-named Longhorn, which is now expected to come out in 2007 or later. Office 12 initially was slated to ship with Longhorn, but the next-generation Windows platform slipped and Office didn’t, according to one insider. “The Office team is disciplined. They nail down their feature set, set a schedule and usually hit it,” the insider said.

Read all this in the context of my previous posts on Choosing an office suite

Atlantic published an article about the tools and techniques promoted by David Allen the author of the book Getting things done, which I read a few months back.  I liked the book and gave it a quick review here.  However for a better introduction its a good idea to read the article.  I have repeated a small snipit of it here to get you started.

The doctrine that inspires this devotion starts with the idea that the difference between done and undone tasks is more stress-inducing than most people recognize. In earlier times, Allen says, work was more physically exhausting than it is today. But it produced less anxiety; because people could easily tell what they had to do and whether it had been completed. Either the wood was chopped or it was not. The typical modern day, he says, is a fog of constantly accumulating open-ended obligations, with little barrier between the personal and the professional and few dear signals that you are actually “done.” E-mail pours in. Hallway conversations end with ‘I’ll get back to you.” The cell phone rings. The newspaper tells you about movies you’d like to see, recipes you’d like to try, places you’d like to go. There are countless things that everyone really “should” do more of–exercise, read, spend time with the family, have lunch with a contact, be “better” at work. The modern condition is to be overwhelmed–and, according to Allen, to feel not just tired but chronically anxious, because so many things you have at some level committed to do never get done.

The anxiety is compounded, he says, by a foible of the human mind: it can’t remember, and it can’t forget. No one can possibly remember all the promises, deadlines, and other “shoulds” of personal and occupational life. The proof is the need for datebooks. No sane person tries to keep all future meetings in his or her head. But, perversely, the brain also can’t forget; at some deep and not very efficient level it is always stewing about the things you should have done but haven’t, and it tends to remind you of them at the worst time–typically, 3:00 A.M. A vague but powerful awareness of all these uncompleted promises, or “open loops,” is what Allen sees as the basic source of work-related stress, Again, datebooks illustrate the point. People complain about their schedules, but they rarely wake up at night worrying that they won’t remember to go to the airport on the right day. That is because they trust their datebooks and trust themselves to look at their datebooks regularly.

If you are serious about personal productivity though then you could do worse than invest some time reviewing the discussion forum that David Allen’s company hosts.  There is a very active debate in this forum and its quality stuff.

HP have posted a short article on the role of the Architect throughout the IT project lifecycle.  I like the perspecitive as often the dominant role of the Architect is the “design” role, and this is the one that is most repeatable and easy to capture in methodology.  The following picture provides a summary of the idea, with the yellow boxes representing the roles, and the ovals the phases.

The whole article is available here. but this is a summary:

The following sections map the the responsibilities and associated skills and capabilities of architects onto the Architecting Process. The responsibilities may fall to a single architect, or be distributed among a team of architects. Thus, someone on the team must play the role of leader who is empowered by management and the architecting team to make decisions that stick (the “architect with a baseball bat”). Others on the team may take on implementation responsibilities during prototyping or technology evaluation. If the architecting team has a project manager, that person may take on much of the “up and out” communication responsibility in addition to handling the personnel interface. Nonetheless, the lead architect must have a good proportion of the skills and capabilities identified below. The rest of the team must be good collaborators, modelers and system thinkers with significant experience in the domain.