Memcached - a great piece of xplatform caching kit

Posted by Ken | Filed under , ,

Have just been reading up on a distributed cache api, Memcached.   Memcached works like distributed shared memory, with the address space being spread across all nodes in the system, i.e. a cached item is not replicated to each and every node.  Instead, memcache allows you to share spare RAM across a cluster of web servers.  For example, if you configure three servers A, B and C and then add and a data item to the memcache on server B, it doesn't get replicated to servers A and C.  Instead, the memcache on server B must be accessed from A and C.  

This caching api runs as a daemon / windows service, with a variety of clients for different platforms.  Though not for all applications (here's a list of reasons why you shouldn't use Memcached), the benefits are as follows: 

  • Reduce database load for high-traffic content-driven sites
  • Reduce latency associated with DB calls
  • To cache computed objects to relieve CPU load
  • Cache small but frequently accessed files
  • Cache results from web services / RSS feeds

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Nice quote from John Leckie

Posted by Ken | Filed under ,

John Leckie is a British record producer, famous for his work with XTC, Radiohead, The Stone Roses and Kula Shaker.

Here's some choice quotes from his MixMag interview, back in 2001, discussing recording Radiohead's The Bends...

People ask me all kinds of things about that album, like, “How do you get that vocal sound?” It's easy. Just get a good singer with a good voice, put the singer in front of a microphone and it sounds good. Now, if you have a bad singer, even if you've worked like mad to make him sound good, no one takes any notice. But if you have a good singer, you just record the vocal flat, and you don't even compress it or limit it. You just put the fader up, and he sings the song first go. Everyone says, “Ah, the producer's great! Incredible production, man!” And it's just because the guy's a good singer!

The same with guitar sounds. I mean, one of the things when we started the record and they wanted to find a new sound was we got every type of guitar in Gibsons, Fenders, Danelectros and all sorts of weird amps, like Soldano amps. And, after three weeks fiddling around, we came right back to where we started, which was just a Fender Twin and a Telecaster.

So... keep it simple, capture the best performance you can. Guitarists - set the intonation and tune the fucker!

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Notes on Domain Driven Design

Posted by Ken | Filed under

At the moment I’m reading Eric Evans’s Book “Domain-Driven Design: Tackling Complexity in the Heart of Software [published by Addison-Wesley, ISBN: 0-321-12521-5].  Although the book was written in 2004, many of his observations and motivations for DDD still ring true - little having changed in the world of object-oriented design.  The book is really the first place to look if you're interested in learning DDD. 

What I like - DDD (done well) is an effective way to manage and communicate a complex set of business rules.  Keep it simple, use objects the way the inventors intended!

What I don't like - DDD has quite a few guiding principles, increasing the learning curve.  This is not in itself a great issue, however proponents of DDD can be very dogmatic (aka anal) about how to apply the principles wrt entities, value objects, service locator patterns and so on.  This drives new programmers away in droves!

I'm lucky enough to have had first hand experience of domain driven design -- back in 2005 I worked on an ASP.NET insurance project for Toyota Financial Services (an online origination system used throughout Toyota's Australian dealerships).  Because the dev leads were crossing over from the Java community, many of the techniques employed (.NET 2.0 in Beta, AJAX, nHibernate, Spring.NET) were 'bleeding edge' at the time - certainly a while before ALT.NET was dreamed up.  However, the simplest piece of the puzzle was indeed the domain model.  It's a very natural piece of an OO puzzle, and it proved to be key to the project's success. 

.NET is an object-oriented language, or did you forget? 
So why is DDD not more more widely used in .NET projects?  The main reason seems to be that .NET programmers become very dependent upon visual tooling.  Because of this, the more esoteric aspects of object-oriented design (like design patterns, dependency injection or ORMs) are quite alien to .NET programmers.  Thankfully this is something Microsoft is trying hard to put right, but you will still find experienced .NET guys routinely coupling application logic directly to underlying infrastructure, whether that be a Windows form control or a stored procedure.   This is a great approach for a small project - simple always gets the best results - but the outcome is defect hell in enterprise development.  

Lessons from recent projects 
A few years ago I joined a banking project mid way through -- it took us twelve weeks to move the application a few centimetres -- it was plain that the complexity was killing us, because the lead devs had put all their business logic in the UI!  A developer fixing one bug would introduce another, and had no means to build any tests in isolation, because the logic was all in the UI.  The team leads -- the guys who had designed the f***ed architecture in the first place -- blamed the devs!  I'm glad I turned up late to that party.

Twelve months later, we're now in the process of re-running a similar project for wholesale share trading.  Thus far we're about to go into production, in around half the time, with one third of the defects, with a smaller team.  Why?  Because we applied some up-front design, rather than diving in to ad hoc development.  By applying some basic principles of service architecture (like data encasulation), employing a simple domain model (objects with inbuilt behaviour, not datasets with extrinsic (repeated) behaviour) as well as techniques such as code generation to handle database persistence, we've achieved one thing: we removed complex logic from the UI and stored procedures.  In other words, by architecting the application in order to put business logic in the domain objects, we get a much better end result.  In fact, the difference is night and day -- let's hope they realise why!! 

Thats my rant on DDD over for today... below are a few snippets of Evans' book, with my throughts attached...

“We, in the enterprise development community, especially the web development community, have been tainted by years of hype that took us away from proper object oriented software development. In the Java community, good domain modeling was lost in the hype of EJB and the container/component models of 1999-2004.”

The same can be said of .NET software, which has been ruled by the obsession with Visual Studio designer-driven drag-and-drop integration with SQL Server.  Just the same way as you would expect a strong HTML coder never to touch the designer in Visual Studio (it produces the worst HTML known to mankind), you wouldn't expect someone with a knowledge of OOP to write data access code using a designer and datasets.

Until recently, Microsoft had not rushed to develop an enterprise level ORM solution – with third party ORMs filling this space.  Now we have Linq to SQL and (coming soon) the ADO.NET Entity Framework (so far universally hated by the ALT.NET crowd).

"…the community is lacking a clear vision for how to apply object orientation on an enterprise scale, which is why I think DDD is important”

This is still true today, and arugably especially so of Microsoft developers, who tend to prefer data-centric (dataset driven) architectures that come out of the box in Visual Studio.  It seems that nHibernate scares them, despite the fact that Hibernate has been commonplace in Java enterprise projects for some years now...

Some of my projects were very successful; a few failed. A feature common to the success was a rich domain model that evolved through iterations of design and became part of the fabric of the project."

Yes, but is an extensive domain model necessary for all applications?  In some ways it's an additional complexity that one should only attempt on a large project -- where you have resources to manage it properly?  Often the business wants the simplest solution that will generate positive ROI -- not something that will become a core app for the next 20 years....

What is a domain model?

A domain model is business process knowledge, captured.  It results in -

  • a framework for making design decisions - the building blocks of a model-driven design are best practices and shared design patterns (e.g. layered architecture, Factories, Repositories)
  • a common technical vocabulary – the fundamental goal of DDD is to use the domain model to drive communication and design

Approach to design is critical

"It is the approach to design that largely determines how complex software can become"

"The most significant complexity of many applications is not technical.  It is the domain itself, the activity or business of the user… a successful design must systematically deal with this central aspect of the software." 

I agree that the design must address the complexities of the business domain, however in incremental development the business requirement usually isn't fully understood at the outset.  Business requirements usually get more complex as the project progresses, and without the appropriate management, the code usually gets in an unreal mess :)   Most ad-hoc incremental development development tries to deliver by tweak-tweak tweaking, instead of revisting the design afresh - which of course just incurs a design debt.  There comes a point where the code's 'logical infrastructure' isn't capable of being extended to support the complexities of the business logic required.

"XP works best for developers with a sharp design sense"

That suggests that DDD has a talent dimension -- accordingly it's not something you should attempt to use with a team of junior developers who've not worked on a domain model project previously.  All we need is well structured code with enough tests to allow it to be safely refactored or completely redesigned when the time comes - full lifecycle testing.  However, the issue with this is that you need a strong programmer

"… with typical design approaches, the code and documents don't express this hard-earned knowledge in a usable form, so when the oral tradition is interrupted for any reason, the knowledge is lost."

Projects with technological debt

"Developers were flying by the seat of their pants, but this didn't hinder them because simple software can be written with little attention to design.  As a result of this initial success,expectations for future development were sky high!"

Technological debt results in

  • lack of a domain model
  • Lack of a consistent, common language
  • Unstructured design
  • Poor architecture - meaning refactoring / retrospectives are impossible to attempt

We’ve all worked on new projects whose codebase has to be treated like legacy code…

"... their first release had ossified prematurely into a high maintenance legacy."

lol!  How true is that!

"Early in my career, I was fortunate to end up on a project that did emphasise domain design"

"This upward trajectory was directly attributable to an incisive domain model, repeatedly refined and expressed in code"

"As the team gained new insight into the domain, the model deepened. The quality of communication improved not only among developers, but also between developers and domain experts"

When domain model fails - communication disconnects such as this are commonplace in Enterprise development:

"a poorly chosen separation of developer roles disconnected modeling from implementation, so that design did not reflect the deep analysis that was going on" 

I think architecture is the main culprit – if you’re obliged to apply dataset / DTO thinking across the board, this then prevents developers from achieving any awareness of object oriented design:

Developers often have  "no awareness of the informal body of style and technique for creating model based objects that also function as practical, running software."

The conclusion is simple: medium to large projects require a well thought out up-front design approach, which is ideally lightweight, as in Agile projects.  You need decent devs to lay the groundwork before you try to scale out quickly - some basic architectural guidance is a must, together with ongoing mentoring in object oriented design.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Emma Pollock is the bees knees!

Posted by Ken | Filed under , , ,

For the last few months I've been listening to Emma Pollock's 2007 album "Watch the Fireworks" ... if you haven't heard her, please note that the track "Fortune" is a work of indie genius (amazing backing vocals!) "Limbs" is similarly a work of Glaswegian art par excellance... beautiful!  Roddy Frame would be jealous ; )

Since moving to Aus in 2002 I've become a fan of the Delgados (a great shame they split), and "Universal Audio" is awesome stuff.  One stand out (there are a few) is Pollock's "The City Consumes Us" -- this floors me on approx seventeen different levels.  Wonderful lyrics, and oh What a chorus...!

As they say in Glasgow, this girl is absolutely fucking brilliant.  Go and check out Emma's stuff now at http://www.emmapollock.com/

 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Using Code Generation to underpin Test First Domain Driven Development

Posted by Ken | Filed under ,

At the moment I'm reading Gerard Mezaro's book xUnit Test Patterns. It has some great ideas on how to manage and test large amounts of code without creating huge maintenance overheads.  

A commonly used pattern in application development is to apply 'white box' functional tests against a service facade, and have the underlying business logic call all the way down to the database tier, perhaps using a 'sandbox' database reconstructed on each build.  Some people call these tests 'story tests', because a use-case or user story is being functionally tested end to end.  

The advantages of this approach are as follows: 

  • Automated tests allow you to verify use case functionality 
  • The whole stack is tested, giving you a high degree of confidence the application will behave correctly 
  • Because automated tests are mapped directly to use cases, defects can be traced back to specific use cases, meaning system documentation can remain in sync throughout the entire lifecycle of the application

However, testing the 'full stack' brings with it various disadvantages -

  • there is additional development / maintenance cost of setting up and tearing down the underlying database state needed to run each test; because multiple tests may share the same setup, it becomes very easy to construct brittle tests which have underlying data dependencies 
  • because integration tests run out of process, these tests run much slower than in-process unit tests
  • when integration tests fail there is poor defect localization - it may take some time debugging to ascertain whether the issue lies in C# code, underlying data or in stored procedures
  • the additional setup and environmental dependencies ensure we result in overly complex tests which are very difficult to maintain 
What would be an improvement to this test architecture?    

A testable Application Architecture to Maximize investment in code

The use of a Domain Driven Design Repository pattern allows us to disconnect business logic tests from database code.   An architecture might be as follows: 

  • UI - MVP Pattern (hand written UI logic tests)
  • Application Tier - a rich Domain Model whose tests use an abstract data model based on a Repository pattern
  • Entity Service Tier - ORM or manual adapter (fully code generated CRUD tests using automatic teardown with database rollback)
The combination of these approaches give an interesting architecture - because there are very few cross-layer tests, all tests run within their own layer.  Although this means all  dependencies must be fully mocked, perhaps resulting in additional code within the service tier, the absence of integration testing means that business tests will run very fast.  All you need to do it create an object (POCO = Plain Old CLI Object), populate it and have your mock repository return the object to the calling test code.   
 
More importantly the amount of functional test code required to set up and tear down test fixture database state is vastly reduced.  Where does all the code go?  Well, apart from a few integration tests that could run end-to-end (to verify environmental items such as the the correct configuration for each database user profile), everything else (CRUD, mapping to domain entities, tests for CRUD) would be code generated into its own assemblies. 

In using the ORM, we're asking you to trust that its generated tests can cover basic CRUD operations 100%.  You have to trust that your data repository will give you the business objects your application expects, and that you're safe to test from there up.  You can test business logic only, do no integration testing. 

A big leap of faith for some perhaps?  What can go wrong? 

  • DAL code should always be linear - get / entities and make sure all fields are correctly populated. You can't let your UI handle nulls - your DAL should return properly constituted objects
  • If you write your DAL manually, you can create integration tests there to get / set / search objects.  By testing downwards, from your DAL code to your database, you can use automatic teardown by rolling back the database transaction after each test.
CodeSmith templates such as NetTiers generate this code very efficiently, and their templates are easily customisable if you know a little ASP.NET. 

 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Agile Model Driven Design vs Domain Driven Design

Posted by Ken | Filed under ,

Scott Ambler has written a great selection of articles on Agile development at http://www.agilemodeling.com/

One article that caught my eye was Scott Ambler - Agile Model Driven Development (AMDD).  The thought here is that Agile development seems to retrospectively lay claim to anything that is not 'pure waterfall'.  I think that's taking a draconian view of traditional development methods, which were never as inflexible or as set in stone as modern devs would have you believe.  It seems to me that a great deal of what Agile professes as novel was suggested decades ago, as part of the RAD movement - check out Barry Boehm's Software Engineering Economics.

As the name implies, AMDD is the agile version of Model Driven Development (MDD). MDD is an approach to software development where extensive models are created before source code is written. 

I'm not sure this is correct of DDD: "extensive models are created before source code is written."  The whole point is that a model should be allowed to evolve - implicit in any evolution is the presumption that you cannot possibly evolve faster than your requirements are known / understood...

The difference with AMDD is that instead of creating extensive models before writing source code you instead create agile models which are just barely good enough that drive your overall development efforts. 

This is the essence of DDD.  In whatever form you see it - this is not unique to Agile - it's simply an iterative methodology.

 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Ten years of Test Driven Development

Posted by Ken | Filed under

Steve Freeman posted a timeline listing the last ten years of TDD http://c2.com/cgi/wiki?TenYearsOfTestDrivenDevelopment

The post contains some great links to resources for Test Driven Development! 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

MySpace customisation

Posted by Ken | Filed under , , , ,

I've dabbled recently with MySpace customisation, putting together sites for Sydney based songwriters Dave Blennerhassett and Gene Maynard.  For those of you interested in doing your own thing, John Pospisil's book is a great all-round intro.

Aside from the realisation that the CSS layout for myspace is a bit of an abomination (why didn't they pay a CSS specialist to do their design work?), in customising MySpace, Firebug is an invaluable tool for identifying the myriad of convoluted styles that may act on any given element.

Last but not least... I haven't yet done any work with the all new v2.0 Profile Editor...but stay tuned!




 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Talk Me Down - Mix from Digital Performer 5

Posted by Ken | Filed under , , ,

As some of you may know, I've recently moved away from the Pro Tools world, despite having spent three years enjoying what Pro Tools has to offer (plus a small stack of cash on the Digidesign certification and training : )

The guys at Conti recently upgraded their LE system to a 48i/o Pyramix system with SSL converters - for quite a bit less than the equivalent HD system would have cost them.  Sounds great and (unlike Digidesign HD) the base system isn't limited to 24 channels of I/O.  

So, given that I no longer needed to support Pro Tools in my home setup, I opted to revert to the same topology I used back in the 90s - i.e. upgrade my ten year old MOTU PCI-324 card to a new (G5 compatible) PCI-424 card, meaning I can now run Digital Performer 5 via my dusty old MOTU 2408 and Ramsa digital desk.  

I'm still a huge fan of what Pro Tools has to offer, especially for "mix inside the box" applications, however the LE interfaces sound generally 'mushy', which is why most serious recordists will avoid the analogue I/O and feed inputs digitally from an external converter. 

Tracks were converted from Pro Tools session format to OMF via DigiTranslator, then imported directly into DP5.  Here's a rough mix of "Talk Me Down" - Talk Me Down - Oct 08 Mix.mp3 (4.89 mb)   

  • The original tracking was performed via Cubase VST24 and Protools LE. 
  • Mix via Digital Performer 5 / some external summing via Ramsa WR-DA7
  • Reason strings / cellos etc
  • Kit is all BFD v 1.5
  • No mastering (just a rough mix) so it's a little quiet (turn it up!)

Enjoy! 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5