Raphael's Blog

Andriod Sonic CD

2012-02-08T09:44:00.001-08:00

Just picked up a copy of Sonic CD for Android. This is an essential purchase for any retro game collector. Given the recent hype surrounding the classic incarnations of Sonic it is no surprise that this fogotten gem has made it on to nearly all the casual gaming platforms going. I was lucky enough to have a copy of this back in the day, but I had no problem picking this up for a few quid.

Nearly every version I have for this game uses the North American sound track. This is a massive shame as the original Japanese soundtrack is gorgeous. Thankfully, the recently released version comes with the Japanese soundtrack as standard.

The touch screen controls are good, if a little hit and miss, so you may want to play this with the keyboard, should your device support one. I am lucky to have a keyboard on both my Android devices, the controls are a standard WASD for direction, with L for action (you only get one button for every action, you dont need any more.).

This version also has a few other refinments, firstly, you can alter the depiction of the spin dash attack. You can choose to use the original spin animation or the version used from Sonic 2 onwards. You also get to choose Tails as a playable character, but unfortunately you are not able to get a Super Tails with all the chaos emeralds.

For the best part of four quid, this is an unbelievably good bargain. A slice of Sonic heritage that is so often missing from Sega collections. I reccomend that you get this game if you have the chance.

More Tablet loveliness

2012-02-08T09:34:00.001-08:00

This week I picked up the UK folio case for my new Lenovo Tablet along with the stylus. The folio case pretty much transforms the ThinkPad tablet into a fully fledged Android latptop. The built in keyboard and optical mouse respond very well indeed.

On its own, the stylus allows you to have precise pen input, the built in handwriting recognition performs a lot better when used with the N-Trig device than your finger alone. Note taking is improved and you can rest your palm on the screen - something that using your finger wouldnt let you do.

The folio case is in an attractive black leather with all the signature colours you would expect from a Lenovo device. The only thing missing is the inclusion of the keyboard light, something that appears on a lot of Lenovo devices.

Access to all the connectors is provided, however the memory card slot door and full sized USB port are covered. This case uses the USB port for attaching the keyboard and optical mouse, so you wont be able to attache your HDD whilst this is in use. Unfortunately, the case doesnt allow for easy use or acces of it either, which is a shame.

The case also obscures the memory card slot door, if only partially. Which means you wont be hot swapping memory cards whilst in use. This is a bit of a pain, but nothing major. Access to the docking connector, power connector, headphone/mic connector and pen storage is catered for.

The mouse works well, another Lenovo signature addition, instead of a trackpad - which would have extended the keyboard further and thus ruin the design, you have an optical mouse in place of the traditional Lenovo nipple/nub (dont know the official name for this). It works as well as you would expect it, the only negative comment I have about it is that the pointer icon itself seems to be a little on the small size.

In my personal opinion, if you have a Lenovo ThinkPad tablet, then getting both the folio case and the stylus is a must. Whilst the screen keyboard on this tablet is easy to use, it does take up a lot of screen real estate, the folio keyboard reclaims this resulting in an end product that is highly adaptable to pretty much all situations you may need a laptop replacement.

Finally, I own an Amiga 1200!

2012-01-24T04:42:00.001-08:00

This is a big deal for me. When I was growing up, I always wanted an Amiga, I think I was intoxicated by the wonderful graphics from the games I watched being played on Gamesmaster. Compared to my Sega Master System, this thing was the mutts nuts. So, when I saw the opportunity to grab an almost mint Amiga 1200 on eBay, I jumped.

I had been after an Amiga 4000, but this was such a good deal I couldnt leave it alone. It even arrived with the original box and manuals. Things like this always make be happy. After some quick research on the Internet, I discovered that there were numerous upgrades available for this computer, including an acellerator board, a PCMCIA networking card and a SCSI interface for a CD ROM. The device I got came with an additional floppy disk drive, which was nice, especially if the internal one decided to give up the ghost.

With the Blizzard board fitted, I have now almost doubled the power of the computer. All I need to do now is fit some form of hard drive and upgrade the ROM's to KickStart 3.1. Once that is done, I will hook it up to the network and see what I can do on the Internet with it, probably some very self-congratulatory posts on FaceBook to announce its arrival on the Internet.

I dont think there are many 1200 specific games available, I think Beneath A Steel Sky is one of them. Hopefully I will be able to track down a copy and give it a whirl.

NEC PC Engine FX

2012-01-23T14:41:00.001-08:00

A couple of months back, I took delivery of my latest console - the PC Engine FX. As far as I know, this was a 32Bit console that was only ever released in Japan, some how, a brand new boxed example managed to end up in Cornwall, thank God. Buying one of these from Japan costs a lot in shipping.

The device itself is a odd one. It is CD only, which isn't surprising, and it is designed in the style of a small tower PC. For some reason though, there a whole bunch of little secret compartments in the main body, I guess that NEC saw this a being a runaway success, so decided to make it extensible (even though by this time, manufacturers should have realised that upgrades like this didn't sell so well). I don't have any games for it yet, a lot of them seem to be JRPG's and my Japanese is non-existent. Once I find a good example of a Shmup or a brawler, I will fire it up and see how its years in deepest darkest Cornwall have gone down.

As far as I know though, this console wasn't even big in Japan... which I think is a bit of a shame really as it quite a charming little device. The joypads feel a bit cheap, but a substantial - and has six buttons, a staple for new consoles in the mid 90's.

I guess I just need to hit YouTube to get some vids of some of the games available for it, then hit eBay. I really must get around to recording some footage of these devices and upload them. I am not very sure how many other people are interested in things like this to be honest, but given the surge in interest in retro gaming recently, it may be good to talk about the 'also rans'.

Me and my new tablet

2012-01-23T14:13:00.001-08:00

A friend of mine recently gave me a new Android tablet, I have had an Android phone for about a year now and have been nothing but impressed with it. That I like Android as a smartphone OS is a big deal for me - I am a reformed Nokia fan. Personally, I think Android turned up just in time, Nokia lost the plot a bit when it came to smartphones and let the iPhone steal a lead. Granted, Nokia had been trying to get tablet computing off the ground for a while (the N710 etc), but like its phone offerings, they seemed to lose touch with their consumer base (note to self, get on eBay to pick up an old Nokia tablet).

As you will see from my blog in previous posts, I love old tech and how it has influenced today's products. For example, the tablet I am currently using to write this post. It uses am Arm CPU, as do a lot of things these days, which of course is related to one of my favourite old school computers - the A7000+. It also uses a touch screen interface, which is normal on every device these days, but this tablet offers stylus input - this is unique I the days of multi touch capacitive screens. It almost seems old fashioned, years back we use to have to make do with a stylus on a resitive screen to have any form of input. These days it almost seems like a backward step to include something like that on a new design. Think Psion Series 7, this tablet I am using, a Lenovo ThinkPad, can use a stylus (unfortunately one doesn't come with it in the box), it can be usedfor regular operations or for handwriting recoginition. Thankfully though, you can also make do with your finger for this - doesn't work so well with my blunt digits though!

Connectivity is wonderful to. I have WiFi and Bluetooth of course, however I also have a mini HDMI out, a full sized SD card slot and a SIM slot (although my model doesn't have a WWAN card - something I plant to remedy soon). Added to this, I also have a full sized USB port. This is primarily used for storage, which is pretty amazing on its own. I hooked up my 1TB USB drive with no problems. However, when I tried this with a wireless USB keyboard it did not work.

One thing I did find annoying was the inability of the Netflix app to work. It consistently complains of no network connectivity even though I have a good WiFi connection. Same goes for the Virgin Media app - a hold over from phone apps maybe? The BBC I player worked wonderfully though, and I was able to install all of my previously purchased apps as well.

One thing I am very keen to explore is the gaming potential of this device. At the end of last year two rather wonderful games were released for Android - Sonic CD and GTA3. I have Sonic CD installed at the moment and I love it. It is one of the games I keep telling myself to pick up for my Mega CD, but never do. With the release of the Wii U later this year, I can finally see why people are making a big fuss about tablet gaming.

Whilst I know this device will improve my productivity, I also know I will use it a lot in my down time. I have some films stacked up on now to watch on the journey to work. Hopefully I will also be able to find a Bluetooth gamepad so I can indulge in some Retro gaming :-)

Accurev 5.3

2011-11-24T04:50:00.000-08:00

Been working with the latest version of Accurev this week.

The latest version now uses POSTGREsql as its back end, replacing the previous DB. Carrying out an upgrade from one to the other will be a big job, however I am kind of looking forward to the challenge.

The testing I have carried out so far seems to show a massive improvement in speed when populating and creating a workspace.

Once my upgrade is done, I plan to hook the whole thing into my continuous delivery platform, more to follow.

The Commodore CDTV

2011-08-24T04:30:00.000-07:00

My latest purchase, a Commodore CDTV has arrived and has been set up for use. Essentially, this is just a tarted up Amiga 600 with a nice, sleek black case and some different input methods.

This purchase was quite a steal, the example I got came complete with a matching black floppy drive as well as a matching black HDD, keyboard and mouse. The CDTV was primarily a CD based console that was designed to fit in with your stereo equipment under the telly. I remember this being advertised by Special Reserve back in the day, but never really knew much about it. Never saw an ad on TV for it, which was kind of odd for a Commodore product. Being one of the earliest CD based systems, it didnt have a motorised CD tray, instead you had to place your CD Rom into a caddy and plug that into the machine. This will seem very odd for people nowadays, especially as we dont really have that many motorised trays about any more. I am not sure as to why Commodore went for this configuration, to keep costs down perhaps?

The joypad that comes with the console is way before its time. It is wireless, something that is the norm now, but in the old days was not. It is kind of similar to the Jaguar joypad, in so far that it has so many buttons, this was primarily down to the fact that the device was pitched as a hybrid console/entertainment device (in a similar way to the CDTV). I haven't been able to find any decent titles for it yet to test out the joypad, but I have been able to boot it up into its computer mode. Apparently, there is a jumper on the main-board that will convert the device into an Amiga 600. I plan to install a switch so that I can try this out. I have a copy of Street Fighter 2 for the Amiga 600 that I am anxious to try out.

Unfortunately though, there doesn't seem to be any way I can hook this device up to the Internet, which I think is a bit of a shame.

Just fixed my Jaguar and Jaguar CD

2011-07-24T04:14:00.000-07:00

Good news, my bargain basement Jaguar and CD expansion is now working.

I was pretty lucky when I managed to pick both these consoles up at once from eBay for just under £80, anyone who is in to consoles knows that the Jaguar CD normally goes for a few hundred pounds at least. It arrived a few months back with no PSU's, AV cable, joypad or games. A quick search on eBay landed me some PSU's, a S-Video cable and a joypad. More recently, I picked up a copy of the seminal Alien VS Predator.

If I had been given one of these, with Alien VS Predator, back in 1994, I would have thought it was the coolest thing ever. These days though, with all the wonderful HD games we get on modern consoles, I cant help but think that this is showing its age quite a bit. One thing I do want to do though, is defend the much hated on joypad - it really isn't that bad.

You will see a lot of people claiming that this is the worst joypad ever created. Now that I have used one, I cant help but think the haters may never have actually used one. Yes, it is big and yes it has loads of buttons - but is it any different to a controller on a modern console? Both the Xbox 360 and PS3 feature controllers with up to 14 buttons (if you include the d-pad), 15 on the 360 controller if you include the back button (a la Halo). On its own, the Jaguar controller has 15 input buttons.

A popular topic for debate with the controller is the addition of that keypad, people ridiculed it at the time as pointless. But, if you look back into gaming history, you will find that there are numerous successful consoles that have implemented the same button configuration. It wasn't really pointless, but instead a flashback to consoles from the 1980's. The worst thing about the controller is the d-pad - it is horrible. Very stiff and very clunky, when you compare it to a Sega 6-button pad, it is a complete fail.

Overall, I am pleased that I got this console up and running, especially for such a low price, but I don't think I will be playing it much in the future. I plan to complete Alien VS Predator and I also plan to purchase Kasumi Ninja for nothing more than the sheer hilarity of the game. But other than that, it iwill just be an interesting, albeit fully functional, curio. One thing I definitely keep my eyes out for is the official 18 button controller, it has an extra three large red buttons, probably for aspirations of getting a Street Fighter onto the console.

Getting the Primary Key Column Name in SQL Server 2008 R2

2011-05-31T06:11:00.000-07:00

This might be something that is obvious to most people working in SQL Server, but for me this was a new one.

I needed to get the column name for the primary key in a table - I am working on a migration just now (no great surprise there ;)) and I needed to get this column name in order to carry out some double posting (once in the new system and next in the old system).

It was surprisingly easy to do this

 SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE
WHERE TABLE_NAME = ''

This simple piece of code gets the name of the column that is your primary key. Short and simple and a new one for myself.

Apple Bandai Pippin

2011-04-05T07:30:00.001-07:00

Just took delivery of my Apple Bandai Pippin this afternoon. This was a console that I purchased back in February from Ebay. It came from Japan, so the shipping prices were exorbitant for airmail, leaving me the option of surface mail.

It isn't brand new, like I expected it to be, but it is in very good condition for a second hand device that has travelled most of the world to get to me. I also had to pay £20 in customs fees, which was a bit of a pain.

The device itself was released in the late 90's and was designed by Apple along with Bandai to represent a significant threat to the dominant consoles of the era. I remember this console being announced in a UK video games mag - I forget which one just now, possibly Edge. Compared to the announcements made by Sega for the Saturn or Nintendo for the N64 which took up multiple pages, the humble Pippin had to make to do with a tiny quarter of a page at the back of the magazine.

This console is pretty notable for the time, like the Sega Dreamcast which was released around about the same time, it supports VGA. For the Dreamcast you needed a VGA box and I don't think every game supported this technology. The back of the Pippin is pretty well equipped, it has the regular AV connections as well as a VGA port. There is also a switch that lets you select VGA, PAL or NTSC as the output. There is also a connection for a keyboard as well as a pstn modem. This starts to give a larger picture concerning Apples intention for this device. Pretty early on in its development, Apple decided they couldn't compete against the Playstation or Nintendo N64 (I think they were the dominant force at the end of the 1990's), so they took the brave step of marketing this as an internet capable machine. Apple also asked for a very large amount of money for the Pippin, which has hardly any games. Apple took the position that it was a cut down Mac - which it is, but also tried to sell the pick up and play-a-bility of the device, which is fine.

Like the Dreamcast, the Pippin uses a proprietary operating system - Apple OS 7.3 I think. On powering on the console, its Mac heritage is plain to see. The console comes with three cd-rom's, one contains office utilities like Word, Paint and Email, another comes with Internet Explorer 3 for the Mac. Unfortunately, I don't read Japanese so I am not to sure what the third disc holds.

Again, like the Dreamcast, the Pippin supports a modem. Unlike the Dreamcast, you get the modem in the box with the console, and can get up and running on the internet in next to no time, as long as you have access to a dialup service. The colour scheme and button layout on the joypad is also very similar to the Dreamcast and PSX of the time. One thing notable about the joypad is that it is one of the best joypads I have every held. It has a total of eleven buttons, four buttons on the right hand side, two shoulder buttons and three buttons towards the base of the controller. One really nice touch is the addition of a trackball, this makes navigating the menus in a PC-like fashion a breeze. There are also expansion ports on the underside of the device, however I have no idea what they were for.

This is a pretty expensive machine to pick up, mine cost over £300, although at the time of writing, there were quite a few sub-£200 versions for sale on Ebay. If you do end up buying one from Japan, which is most likely, then you need to factor in the shipping fees. The console is quite heavy making airmail very expensive (probably half the cost of the console again).

I am not sure as to why the console never took off, perhaps due to the high price and the lack of decent third party games were the biggest contributors. Or maybe it was down to Apple not knowing how market the device. I think possibly this is more like it, Apple didn't sell it as a video game console, but they didn't really try and sell it as a Mac replacement either. One thing it definitely was, and that is ahead of its time. Just like the Dreamcast, the Pippin offers functionality we would expect to see in a console today - internet connectivity, rich media etc.. Put it next to the original model of the Xbox 360 and you would be hard pressed to say that one device was released at the end of the twentieth century.

Overall, I am pretty pleased with this purchase. This is one of three expensive consoles I wanted to get, so this has been successfully crossed off my list. Now, I will scour the Internet and Ebay in the hope of finding a game or two for this console and wonder why this failed so miserably, given Apple then went on to sell things like the iPod, managing to convince everyone in the world + dog that they needed one...

The Nintendo 3DS and how I fell in love with the DS all over again.

2011-03-30T11:48:00.000-07:00

I had a 3DS on pre-order at Game for a very long time. As soon as the 25th of March hit, I picked up my new 3DS and 3D Street Fighter IV on the way into work. I cant say I was blown away with the 3D feature - it is pretty cool, but instead I found the bundled apps and games to be slightly more engaging, and definitely highlighted Nintendo's position as an innovator.

Playing the latest iteration of Street Fighter on a portable console is novel enough, playing it 3D just makes you smile like a little kid. However the Augmented Reality games that come with the 3DS make you really step back for a bit an take stock of this new device. By simply placing a card on my desk and pointing my 3DS at it, dragons started to claw their way out of hole dug out of what looks to be my desk. No, I wasn't on any drugs. One of the bundled games relies on AR cards to unlock some functionality. Whilst this will not be everyone's cup of tea, it does highlight the fact that Nintendo is one of video gamings greatest innovators.

I am not sure how much of a success the 3DS will be when compared to the DS, that console literally printed money for Nintendo. But one thing the 3DS has definitely done for me is play almost all of my DS games again. That means Mario Kart, New Super Mario Bros. and, one of my all time favourites, 42 All Time Classic Games. And that is fine for me, until they release the 3DS versions of course ;)

Working with disk resident files .Net

2010-12-05T13:15:00.000-08:00

I dont often find myself working a lot with flat or delimited files. I either interact with .config files or just with very, very simple text files. However, recently I have found myself dealing with some rather complex delimited files.

When I started working on these files I thought to myself that I would need to develop some code that would do everything I wanted in one fell swoop. However, there are a number of viable alternatives out there that will help you a out a great deal.

Over the last week, tab delimited files were my main adversary and after some pointless efforts on my part to create a robust and useful file handling API, I came across FileHelpers. For now, I am just interested in the ability FileHelpers has when it comes to parsing de-limited files, this whole API is now embedded in the code I am writing though. It struck me that there is a lot on offer from this open source utility that I couldn't ignore.

Take a look at the examples on show in the documentation, there is so much there to make use of it seemed natural to include the API in my work. One of the very cool features I will be looking into during the new year is ability to diff de-limited files. I know there are a lot of tools and patterns out there I could use, but why should I go and make something of my own when this is now on my doorstep?

I urge anyone working in .Net who has to deal with complex de-limited files to take a look at this library, you will not be wasting your time.

Retro Gaming Consoles

2010-11-05T16:25:00.000-07:00

Time for something a little light-hearted I think. I love video games consoles, have done since I first saw a SNES in Currys when I was about 13 years old. At the time, my parents were a little hard up so I wasn't able to get something as swanky as the SNES or the MegaDrive at the time. Instead I got given a Master System - this was my first introduction to console gaming.

I loved my MasterSytem, it was the original model with both cartridge and card slot for games. The built in game was Maze, which I actually played with regular frequency. After a few years I was able to get my hands on a MegaDrive, to me at the time this represented the pinnacle of modern video gaming. I was so pleased that my parents decided to get me these consoles, it was a rare treat for me when growing up to have something this fun to play with.

As time went by, I became a steady Sega fanboy, I was given a Mega-CD as a Christmas present once and not long after that, after working for it relentlessly all summer as a Barrow boy on the local Butlins, I was able to get the 32X. Now of course, this was also the time where things like the Playstation and Sega Saturn were being touted for release. This didn't phase me one bit, I could proudly say to my peers at school that I already had a next-gen, 32bit games console in my house right now. Of course, as the 32X bombed out fairly quickly a lot of my compatriots heaped scorn on me. This didn't really upset me that much, there was no way I could get a Playstation or a Saturn, they were way to expensive for me. But this didn't stop me from reading all about the new machines that were on the way. Of course, after a while, my brother was given a Playstation for a birthday present I think, along with a copy of Metal Gear Solid (the limited edition one with the dog tags etc). I think I had started to go to college my this time, so on one of my off days I picked it up and gave it a play. This was the moment I fell in love with Metal Gear Solid as well as the Playstation.

As the years went by, and I started to earn more money as a post-graduate, I was able to settle my schoolboy attraction to consoles I read about in my early teens. Even thought I have a strong home computer background, I have always had, and always will have a love for gaming consoles. Right now I have a fairly good collection going, starting with the Sega MasterSystem (the same one I was given - it still works today) and going up to the Playstation 3 (the one that can play PS2 games). Recently, I picked up a Goldstar 3DO, this is a magnificent console, I recall reading about it being launched when I was you, but then for some reason people stopped writing about it. There was, for a brief time, a number of articles about the 3DO M2, but this never seamed to get released at all.

The console itself is pretty amazing if you take into account that it was released in 1993. Mine came packaged with Starfighter, which is a truly wonderful game to play. I was attracted to this game based on the fact that it had started off its development as a game for the Acorn Archimedes (and we know just how much I love Acorn Kit). Strangely, the console only has one controller port on the box, additional controllers are plugged into each other, daisychaning them. I am not sure as to just how many can be connected. Another design feature of the controllers was that you could plug 3.5mm headphones into them. The controller also has its own volume control. Now, I don't think that is something that has been seen on a console until the Xbox 360? I may be wrong.

As it stands just now, I have games consoles in pretty much all the rooms in my house apart from my bedroom and bathroom. I will also continue to pick up nice examples of gaming consoles, I would like to get my hands on a NeoGeo and a WonderMega, but these ones a pretty pricey. Which is odd as you can pick up a Hyper NeoGeo arcade motherboard quite cheaply.

Right now though, I am reliving part of my youth by playing Ravenskull on my Acorn Electron which is hooked up to my LCD TV :)

My Lovely A7000+

2010-09-24T12:15:00.001-07:00

Well, it took a massive effort on my part to get my venerable A7000+ up and running again. The machine itself is about thirteen years old and didn't like booting it's original OS. One of the nice things about the A7000+ and pretty much all of it's relatives is that the operating system itself is stored on ROM. You don't need a hard drive in it all really (but it is better if you do). This box was running RISC OS 3.71 the last time I tried to get it up and running, however after years of being stored in the garage, it got a little temperamental. After some careful refurbishment, I ended up having to get a new OS installed.

There isn't that much development happening for RISC OS machines at all at the moment, I managed to find some RISC OS 4 ROMs on Ebay which I fitted myself, after a mornings worth of trying to remember how the network setup works in RISC OS I finally managed to get it on to the Internet. I had planned to do this post from the A7000+ itself, but the supplied web browser just doesn't like Blogger, which I think is a shame. I have seen better browsers for RISC OS, but unlike mainstream OS's they are not free.

There are lots of cool things you can do with a RISC OS machine on a network, but I din't think there is any jobs going in the field just now. I used to work in a school, which at the time had hundreds of these machines. Unfortunately, by this point (2001) there was very little call for the use of RISC OS itself, instead we implemented a Citrix server to provide a Windows session directly to the box. It was a sad thing for me to see really, my first ever computer was an Acorn Election (more posts to come on that one I think) and I grew up using the Acorn Archimedes. the A3000

Old Tech Is Great

2010-09-07T01:30:00.000-07:00

I have been struggling for a while now from the lack of a decent smartphone. I have been using Nokia phones since 2006, prior to that I used Ericsson devices. Now I am stuck with a pointlessly tiny touch screen phone that has no wifi or decent calendar.

So, how am I to cope when I see all these grinning people with shiny new Android devices? Not to mention the fanboys exclaiming madly about how the iPhone 4 is the second coming...

What do I do then? Well, I turn to a gadget I bought around ten years ago to carry my calendar and contacts with me. This device is the Psion Series 5. I have blogged about old tech in the past, I have an almost working Acorn A7000+, an Acorn Electron as well as a few old consoles. My A7000+ is completely functional, running RISC OS 3.7 at the moment, but I am having trouble getting it connected to the Internet, anyway, that's a post for another day...

So, this Psion device. I don't think many people will remember them all that much, as far as I can recall they were never massive hits with the consumer, despite getting very good reviews. When I first got mine, they were still relatively expensive second hand, so I got myself two damaged ones and took them apart to make one working version. However, back in those days I didn't really need an organiser like that - it wasn't like I needed to keep a calendar of things to do, meetings and peoples contacts. However nowadays I do. On my present phone, I cant do this very easily, the only other device I can use for a calendar is my iPod Touch - but Apple seem to have nobbled first gen owners who don't want to upgrade iOS (and why should I? It is mainly updates for the next gen models - not mine). So, I have gone retro geek chic and pressganged my old Series 5 into action for the first time in a decade (alomost).

When I first started playing with them all those years ago, I found it very hard to get it speaking to Windows XP Pro. I am using Windows 7 Ultimate now, so I was a little concerned that the Psi-Win suite that comes with the device would not work. However, after a very brief install period it all works fine. Next on the list was the serial connection, the Series 5 doesn't use USB, it has either an IR port or mini RS232 jack for connectivity. Again, back in the day of Windows XP this was troublesome for me, but again on Windows 7 it works like a dream. Now I can sync my 10 year old Psion Series 5 with Outlook 2010 with no problems at all!

Another interesting thing about this device is it's heritage. The Psion Series 5 runs an operating system called EPOC - this was the direct ancestor of Symbian. If you look at older smart phones like the Ericsson R380 World Phone, you will notice that the interface is almost identical to that of a Psion Series 5. The same thing can be said about the Nokia 9210i as well, the UI is almost identical to EPOC, right down to the soft buttons on the side of the screen (on a Psion, these were either touch sensitive or a soft menu button).

In fact, Ericsson made their own version of the Psion Series 5MX (a slightly updated version) to work seamlessly with Ericsson phones called the Ericsson MC128. I haven't seen any of these devices at all on Ebay, if I ever did I would snap one up in an instant as I have an Ericsson R320 smartphone (count afford the R380 when it came out).

Anyway, the software that Psion created, EPOC, gave way to Symbian. But there is one interesting thing about this OS. It is very similar to that of RISC OS, the two are not directly related. I know that a RISC OS/EPOC Palmtop device was planned, it was called the Osiris but I don't ever remember it getting released.

The only downside I can see with this device these days is the monochrome screen and the lack of a decent messaging suite straight out of the box. I think the Series 5 MX and Series 5 MX Pro addressed these problems though.

So, lets take a look at my Series 5 in it's glory:

The connector at the back is the mini RS232 jack, on the left is a power cable (I have never seen an official Psion power cable for these, so I am actually using a Nokia ACP-12X charger). The Series 5 is pretty good on batteries, but if you are using cheap rechargeable ones like me, it will gobble them up in seconds.

Now for the device opened:

The Series 5 has a very clever way of providing a decent keyboard for typing, when you open it, the keyboard slides out towards you, pushing the screen back to a decent angle. I think this is one of the most ingenious designs ever for a PDA/Netbook.. You can see under the screen and on the left of the screen there is a silver strip. This is touch sensitive and will open up any of the built in apps. You can also control the screen etc from the left hand controls. On the right, there are some touch sensitive soft menu items, such as the control panel.

Now, underneath the device:

I included this shot to show you the last little surprise this device has. You can just about make out that there is a small black panel there - this clever little thing actually hides the controls for a dictation mode the Series 5 offers. With the device closed, you can actually slide this back and record what ever voice memo you need. If you open the device up, you will get this recording presented to you for playback. I am not sure how many people actually use voice memos these days, but you can see that the Series 5 definitely had a lot of thought and effort ploughed into its design.

One final thing about the Series 5, it even comes with it's own development environment installed as standard. The device offers OPL allowing you to create your apps and scripts for this device. It is actually supported in the Sony Ericsson P800/P900 as well as the Nokia Series 80 Commincators (although I do not remember seeing the development editor available in these devices.

So that is it, the Psion Series 5. I think it is as much of an oddity these days as it was when it was first released. It is a fantastic example of British engineering and was definitely ahead of it's time on it's release considering the popularity of netBooks now on the market, not to mention our smartphones...

I will be using mine for a little while yet untill I can track down the last Psion netbook to be released somewhere.

Lego is Great

2010-08-11T08:28:00.000-07:00

Ever since I was a kid I have been interested in building things. I would use anything to hand, building blocks, plasticine and of course Lego!

I think that for most people who are the same age as me and were also Lego fanatics, there are always a few kits that stick in your mind. For me, it was the Mega Core De-Magnetiser, part of Lego's space line up.

At the time, there was no way I could afford it myself, and I didn't have similar pieces to make my own version. Funnily enough, it was a recession then, just like it is now. However, I am older and we have EBay!

I got mine off Ebay, and for something that's almost fourteen years old, it is still in pretty good nick. It really brought my childhood back to me, the only difference is that it took me four and a half hours to build it! I am certain that I would have been able to do it much, much faster when I was small.

Anyway, it was a fun diversion, I even documented the build to prove to others I am not just spouting off.

Now that I think about it a little more, I am pretty sure this kit came out closer to twenty years ago, not fourteen... poor memory must be a sign of old age...>

The chassis:

You don't really get a good idea of how complex this thing actually is. It is essentially a real wheel steering vehicle using split axels - a bit like the current Batmobile, but with added six wheel drive.

There is also a clever crane mechanism that sits on top the thing when it is built. It is one of the rare occasions where a mainstream Lego kit gets to use pieces from Lego Technic. When the build is complete, it is about 17 cm long and about 15 cm high, if you take into account the crane mechanism as well, it ends up as a fairly stout model:

The canopy is a little different to most other Lego Space kits from then as it flips up from the front. If memory serves me correctly, most kits from that time had canopy's that swing upwards.

Either way, I am pleased as punch to finally get one of these kits, I think I waited long enough!!

Continuous Build and Test - Achieving automated deployment

2010-07-05T04:09:00.000-07:00

A few days ago, I posted about how I developed a CI system for an ambitious financial software developer. We had gotten to the point where we had the experience with creating a CI environment but at this point we had outgrown our current build/integration software. Cruisecontrol.Net performed admirably, it had offered us a level of control that we had never seen before. Through the use of Cruisecontrol.Net, we were now able to ensure that all builds would take place the same way based on the requirements of the team working on the code. There was no-one manually running something to build something and forgetting some steps halfway through the process. Development teams were able to see the status of the build they were all working on and received reports and notifications about their code. Management were now able to look at high level reports to see how far along any given project was going. By implementing unit testing during the build process, testing carried out by analysts uncovered fewer and fewer issues and bugs due to this. When it came to deployment, we were also able to run a suite of UI tests post deployment, all carried out based on the spec provided by the development team working on this code.

However, there was still a significant amount of work needed to be done by the test and deployment team at this point. We had a complete hands off build and test system. We could build code and test its logic in one process. We could also test post build using UI test tools such as Watin and Selenium. By this point in time, we were also on the cusp of implementing Quick Test Pro test scripts into the process. There was only really one manual piece of work that we needed to complete in order to have a completely hands of build/test/deployment.

Now, we had a great deal of NAnt scripts that could do pretty much anything we wanted. We were building and running multiple tests, the only thing we could not do at the time was automatically release on a successful build process. We didn't want this to take place in production though, but this would be acceptable for any other environment (including DR). We had a shiny new SCM in place that was fully supported by the build scripts we had created. Right about then, we were also churning out the same amount of code as the main development teams. We worked in c# as well as some other languages like Python and Ruby, whereas the development teams worked exclusively in VB.Net. The last hurdle for us to cross would be the automated deployment after a god build.

It wasn't just the code that the backroom teams were developing that was growing. The company had expanded it's customer base and we were now providing solutions to other global players in the finance sector. The test and deployment teams had already demonstrated that we could work with the development teams to create a build process exclusively for them. In reality, we just a bunch of NAnt scripts and tasks that we used dynamically. The only thing that we needed to do next was to appraise our CI software.

Cruisecontrol.Net is a fantastic tool, but if you work in a company that is providing different solutions to different clients it can be a bit tricky. As the business at large was my teams customer, we quickly found that even though we could put together a CI system for any given team to cater for each spec would mean we would need to implement a Cruisecontrol.Net server per development team. We also found out that Cruisecontrol.Net would be a bit of a nightmare to configure automated deployments. Even though Cruisecontrol.Net could make use of .Net Remoting, the head of infrastructure pointed out (and rightly so) that to achieve an automated deployment using this software would be a security disaster. At the time, Cruisecontrol.Net was a complete CI tool in a box, to implement an automated deployment using Cruisecontrol.Net would have meant we would have had to deploy a Cruisecontrol.Net server to each environment and its selection of application servers. The obvious problem here is that if had have done that, then any malicious users could trigger a build out in the environment itself and possibly substitute our code for theirs. We looked at a way around this, but we could not come up with a solution that infrastructure could support due to security issues - we explained that we could get away with NAnt being out in the environment to handle deployment. Again, and rightly so, infrastructure pointed out that the same problem regarding malicious builds would still exist - basically they said that we were not to have any build tools out in the environments at all. At this juncture in our brainstorming, I pointed out that MSBUILD is already out there and is comparable to NAnt and that it really couldn't be removed as it was needed by .Net! Infrastructure was thankful for this update, but still forbade us from using it for deployments.

Right about now we had two problems on our hands. First up we needed a new CI build tool that could cater for the direction the company was moving towards and secondly, we needed to come up with an automated deployment tool.

We dealt with the CI element first.

We took a look around to see what was on offer. Team Foundation Server could do what we want build wise, but was simply not up to the task as far as our development was concerned. Cruisecontrol.Net was getting left behind with regards to features - the way in which it worked also started to become obsolete for us. We looked at some other CI tools that were available, both the head of development and myself thought we should go for the best of bread solution. This had already been highlighted in our existing setup - there were no tools or frameworks etc that we were using that were entirely dependant on another. From the outset, I wanted each element of the overall system to be as agnostic as possible so that when the time came to change something there would be little impact on our ability to carry our testing and deployments etc. We looked at other tools out there, things like Anthill Pro etc. We ended up going for a tool called TeamCity. It pretty much ticked all the boxes we needed, it was free (up to a certain point) and worked on the server agent architecture - and it was this function that decided it for us.

On adopting TeamCity to be our main CI tool, we asked for one extra server in the build environment (infrastructure did ask why we needed a fifth machine when we were only using one of our existing four...). The new addition was pretty important. Cruisecontrol.Net is a complete build server in a box which means that you need to have it installed on each machine you want to build from (or, at least this is how it worked when we were using it). TeamCity is a little more subtle. It makes use of what are termed as 'build agents'. With a server running constantly listening for build agents, you are able to add as many build servers as you want to cater for each of your projects. For instance, I could have (and have) ran a Linux box to build Mono assemblies. This box would be configured for this type of build only. My TeamCity server is running on a Windows 2008 machine on the network, it has been configured to only allow Mono builds to take place on the box running Linux. With Cruisecontrol.Net, you would have had to configure an instance of Cruisecontrol.Net on each build machine you wanted to build apps from. The TeamCity Agent/Server design is much, much more favourable. It means you can have a relatively cheap box sitting somewhere that is configured to do only one type of build - or in fact be configured to represent a machine out in an environment. You begin to ensure that the code you are building and testing is conforming to the spec laid out for that piece of build. With one server controlling it all and providing valuable MI. When we got to see this in action, we were very pleased. Now we could build for any spec and deliver information on these builds to whomever needed to get feedback.

This again made people happy. It made me happy because it meant that all my guys could simply get on with what they needed to do. The guys writing and maintaining the code we needed for this to take place were not bogged down manually checking that everything had built not only correctly, but also matched the spec for that piece of build. It enabled us to simply agree on a build spec for that development team and implement it. In essence, this was win all over.

As TeamCity uses the server/agent architecture, infrastructure was not adverse to us placing a build agent out in the environments to start working on a delivery system if we needed to do this. There was a lot of work that needed to take place prior to this though, the workstream that would enable a build to be deployed to an environment needed to be bashed out first. For instance, one of our requirements was that we wouldn't release a build to an environment until it has:

Been built successfully.
Passed it's unit tests.
Passed it's UI tests on the build box.

These were pretty simple requirements for ourselves. The majority of the work left to us was to develop the NAnt tasks needed to carry out environmental testing and to create a deployment platform.

There was a pseudo deployment platform in place already, it worked as part of the existing application. it was primarily used to handle some functions out in an environment. Things like pushing assemblies out to application servers and controlling web servers et. I took the decision to move away from this system - I didn't want the build/test/deployment process to be dependant on the actual product we were making.

So, we came up with a prototype platform that implemented .Net Remoting and MSBUILD. There were a few other staging areas as well, like TFTP servers and small services to control things like web-server availability and application servers. As the project for automated deployment went on, we encountered new tools like PowerShell. To me, this was a very useful tool, it meant that I could create c# code that could be accessed directly from within a PowerShell script (we already knew we could script .Net code in NAnt, but we didn't like to do this.). So, MSBUILD gave way for PowerShell, we liked it and so did infrastructure. With these tools in implementation, we needed to define the automated release process. It went a little like this:

Build machine is selected from server farm.
Build machine clears a workspace.
Latest code is updated to the build machine.
Latest code is built.
If the build was successful, unit testing takes place.
On successful unit testing the assemblies are packaged for deployment.
Selected environment would begin a countdown to unavailability.
Once build is packaged and the environment is down for deployment, existing code is uninstalled on target machines.
Latest build is deployed and activated on target environment.
Tests are then carried out in the environment, mainly UI testing.
On successful test completion, all relevant schema and sql is deployed to that environments DB servers.
If the new code fails at all, everything is rolled back and the DB is not touched.

And that was pretty much it. All we needed to do was to come up with some simple .Net classes to handle some of the deployment steps via .Net Remoting. Our PowerShell scripts would then handle the process out in the environment. If it all failed, everything was rolled back prior to releasing to the DB and the team responsible for the build would be notified.

We could have ran a test prior to committal on each developers machine. However, the head of development and myself thought that this could be a little tricky to carry out as there were so many developers working on so many work-streams in so many environments.

When it came to deploying to production, the same scripts and classes were used, but with intervention from someone in the release team to guide it all. So the same process was used, but in such a manner that steps were followed manually.

Just prior to leaving that company, we had created a complete build to release system in just under a year. The system I designed and created with the other guys in the release team (and ultimately the test team when I got promoted) was a world away from the VB scripts we used when I first joined. Now, each customer of the company has its very own build and test specification as well as an automated deployment specification.

Doing all of this would not have been possible if it were not for the availability of the tools at our disposal. There are earlier posts on my blog on how NAnt can be used to create new tasks. Systems like TeamCity allow you to concisely manage all aspects of a development work-stream from end to end. The only thing I never got around to doing for that company was to implement FITNesse.

I continue to praise CI and Agile to my colleagues, when ever I start a new contract, I often ask what build process they have in place. Directly after working for that company, I joined a small team within a huge organisation that needed to create a mini version of my last CI project. I now get asked by developers and employers to explain just how easy it is to implement for their own projects. It astonishes me sometimes to see very complex pieces of software not getting even the simplest unit test - and that has to go through a manual build phase with set specification.

Hyper Geo 64

2010-07-03T06:51:00.000-07:00

As a lot of people who know me will admit to, I am a fan of video games. Pretty much where ever there is a TV in my house, there will be a video game console attached to it.

If you are a guy round about the same age as me, it is possible that you will remember some of the great video game consoles of the 1990's. One that may stick out a bit was the Neo Geo made by SNK. I remember this console having a near mythological impact on video gaming, the games themselves cost almost as much as the console itself, and chances are you never met someone who owned a Neo Geo.

The console itself was pretty much a stripped down version of an arcade machine, this is why the games cost so much money. The AES, which was the home version of the console, provided arcade perfect graphics during a time where people were arguing about whether or not the SNES was superior to the Mega Drive. I think there were a number of attempts by SNK to create a newer, cheaper console for home use - they came up with the Neo Geo CD as well as two handheld consoles.

I think one of the most interesting consoles they came up with was the Hyper Geo 64. From what I can make out, this was SNK's attempt to flirt with 3D graphics. It never made it to the home market though and failed as an arcade machine with only a few games made for it.

Given my interest in consoles, I have embarked on a search to pick up a Hyper Geo 64 and see if I can get it firing. I love old tech like this, so I am hoping that it wont be too expensive to get one.

Continuous Integration and SCM

2010-07-03T03:43:00.000-07:00

Last month, I went on a bit talking about Continuous Integration and how I have developed myself and my skills in relation to this.

I left off by saying that the needs of the company had expanded some what as had the needs of individual development teams. My team was slightly different to the main production and new build teams. The work and code we produced was made to facilitate the development of the applications we offered to our customers. So for us, there was very little in the way of UI development etc. The code that my team developed was primarily to support the rest of the development teams, enabling them to get a birds eye view of development at any one time and across all projects. We didn't just restrict this to management, it would be a bit daft to do that. Pre CI, we often heard the following complaint "we just do not know what is happening". In the build team we were responsible for maintaining all of the build and test environments, infrastructure took care of the guts of them and we maintained the environment based on a development teams specification.

So, to start developing version 2.0 of the CI system, we need to appraise a few new things. First up was our SCM tool. Our current tool was just not geared up to what we wanted it to do. We had been using the same tool for quite some time prior to me joining the company, it was fine in the pre-.Net days where there was only one customer, but as the company grew we needed to get something a little more robust, extensible and modern in. So, one of the first things we did was move away from Surround as our SCM. There were a number of options available to us at this point in time. Team Foundation Server could cater for us at version 1.0, it had a SCM system, it could carry out builds on more than one server etc. The only trouble with it was that it was v1.0 software, even though it was a production release, it was a pain to get it to work. Thankfully, the head of development had looked into this issue quite some time ago with a view to migrating from Surround to a newer tool. His initial efforts at the time met with apathy - there was no need for a new tool as the one they had was fine. So, when the time came the head of development pointed me in the direction of a package called Accurev. By this point in time, I had been made up to the head of deployment and testing, my main aim here was to get a complete CI system, from build to testing logic, to deployment to an environment and then further testing (UI stuff, this was a web app). I took a look at Accurev and liked it pretty much straight away. It was a very modern tool, it allowed you to work under the Agile umbrella of software development and allowed my team to quickly configure code and builds whilst at the same time gave the releasers time to make sure the code we were about to release was the correct code (there was a time prior to Accurev that code hadn't been checked in or re-based properly).

To fit into our CI system, all we needed to do was to create a number of NAnt tasks to work with Accurev, these were nice little tasks to make sure that the code was being updated on the build server. Integrating Accurev, Cruisecontrol.Net, NAnt was pretty simple in the long run, and it let management chill out when we were able to demonstrate how a fairly significant migration could take place with out a lot of drama.

With our new SCM in place and all the developers and other people who needed access to code trained up on how to use it, we were ready for take-off. Took a weekend to export the code we needed from our old SCM into our new one. It didn't take long to script the whole event, but did take a while to get the code from a to b. Once it was done, we were ready. We had a nice new shiny SCM that everyone knew how to work and we were ready to re task the deployment/release team into a new method of CI.

Of course, there were some hiccups along the way. I think the scariest one we had was just after migration, one of my colleagues accidentally deleted some very important code a few days in-front of a major release. bit of panic at the beginning, but we sorted out a major catastrophe fairly quickly. One of the good things that Accurev offers is the fact that it never, ever forgets code. It actually stores all your code ad infinitum, so a quick look through the manual and we were able to bring back all the code we needed after a few bits of code on the command line.

With this migration complete, we were only a few steps away from providing a major update to our CI system.

Unit Testing .Net Automatically

2010-06-10T05:48:00.000-07:00

When I am working on a project, I try to keep the tools we use for building and testing in the same language as any production code uses. So, if I am working on a .Net project, I will endeavour to use .Net wherever possible. The same can be said for things like Java, although I rarely develop in this language, I will try and make sure that the tools used to build, test and deploy the code are based in Java.

Recently though, I have been working on pure c# projects. Since becoming a contract developer, I have encountered a number of lead or senior developers that simply turn around and say "no unit testing, just make it work". It makes me a little sad when I hear comments like this. As far as I am concerned, unit testing is one of the most useful processes a developer can exploit. It cuts down the amount of logic fails in testing and in production. For a relatively short amount of work prior to development, both analysts and developers can agree the unit tests that support the specification - once the test is defined for each unit of code it never has to be changed again (or rarely changed). This cuts down the amount of fails picked up in testing. Some lead developers I have worked with recently have turned down unit testing flat simply because they do not understand what a unit test actually is. You find this situation in teams that are under resourced and over time/budget. The embattled lead developer, possibly dealing with a bit of cynicism, just wants to make the deadline. But, one thing I have seen is that if you take the time to explain the benefits of unit testing, they normally come round and embrace the concept.

These days though, Visual Studio comes along with built in unit testing support. I think it is about time this has happened, even though I am not a fan of what is bundled. My unit testing tool of choice is MbUnit. These days it is part of a much larger test automation platform called Gallio. I like MbUnit for one reason you can test against a DB and roll back any changes once the test is complete, fail or not. I know that typically this is the way you should use unit testing with a DB, but for small things its fine.

Most unit testing tools will come with a test runner that you can use to run your tests, there are also VS add-ins that will do this for you - but I am not going to teach anyone to suck eggs here. Instead, I am going to examine just how unit testing can be automated as part of a larger build process. Some unit testing frameworks will come with an example NAnt or MSBUILD task that can be slotted straight into your build process. There are some out there that will not do this and/or you have the need to execute tests with a test runner outside of NAnt or MSBUILD. Another bit of win for Gallio is that it has very good support for NAnt. There is a task provided by Gallio that allows you to use the test framework automatically, there is a really good example of how this can be done here. The fragment of xml on that site is enough to demonstrate how Gallio can be used with NAnt, however in practise you may find that it is too simplistic, especially for larger builds.

The trick comes down to maintaining your unit tests. IMHO, creating a requirement for naming unit test assemblies is a must, for instance calling your unit tests someting like foo.tests.mbunit is very important for managing the fixtures that will carry out your tests. Personally, I like to provide unit tests within the project it tests. However, I suppose that you could create a separate solution containing all of the test fixtures, I am just of the mind that if you have a project then the fixture should go in that project to maintain encapsulation.

Anyway, regardless of where you keep your fixtures, the naming convention is pretty important as you can make NAnt loop through all assemblies that use a certain naming convention. You could, of course, use NAnt to run an external tool to do this for you etc.

One complaint I used to here from developers when they were just getting used to unit testing was the fact that they needed to run a command on the CLI or use a test runner to prove their code. Again, there is a simple solution to this problem called Test Driven. They provide a fantastic little tool that allows you to test fixtures from within VS itself. I know this may seem like teaching some people to suck eggs but describing this, but there are many new developers who are new to .Net or development in itself who may not know about some of the tools. I am not going to describe any NAnt code here to explain how it can be done. There are lots of NAnt scripts I have written for other bits and pieces I have done in the past, these coupled with the support docs provided by Gallio, NAnt and et al will show you how this can be done.

Back once again with the renegade master

2010-04-14T08:31:00.000-07:00

Been a while since I last posted on here. My aim to put up at least one worthwhile post every month seemed to fall by the wayside.

I still havent setup my old Acorn machine correctly yet, and cunningly my main desktop has decided it doesnt want to speak to my HDD any more.

Good news though, I have a new job that is very interesting to say the least and I have move house.

Bad news, totalled my bike and I have suffered from bad TV and travel karma over the last few weeks. But onwards and upwards! over the next few weeks I intend to put up at least one post concerning the Levenshtein Distance. There is lots of interesting proofs of concept out in the wild at the moment, but there are none concerning an actual useful example. I know that not everyone will be able to implement a really good peice of code to do this (and I doubt anything I produce will be class leading). But this is something I have to work on in my new job, so this will help me figure it out in my head :)

Continuous Intergration

2010-04-03T02:45:00.000-07:00

It's about time I started blogging about CI again, given that I have developed and worked with some of the best CI systems I have had the pleasure to encounter.

When I first got involved with CI it was with Java development. At the start we simply used Ant to carry out our builds and tests on remote servers. As time progressed, we started to look at some of the CI build tools that were available to us. We came across Cruisecontrol, which I think was the tool de jour when it came to build servers. I was naturally impressed with the features offered by Cruisecontrol at the time. We were able to put together a simple CI environment to carry out our builds and/or churns. As time progressed, we looked for ways to integrate Cruisecontrol into our main repository - the company I was working with at the time used Perforce as it's SCM, we also had a tool called the repository that held all of our application meta data (we made mobile phone apps, so the meta data was a collection of definitions per handset and carrier). By the time I left that company, we had almost a complete integration of our build environment with our repository and SCM.

From here, I moved to a .Net software house. I brought with me the concept of Cruisecontrol and CI. My new employers could be best described as a group of energetic younger people who worked on the bleeding edge. This time round, it wasn't mobile phone apps, instead it was the ultra serious realm of financial software. When I joined this company, the guys had just put the finishing touches to a migration to .Net, easy to implement but unforseen issues cropped up - that is a story for another day ;). These guys were up for anything when it came to CI and automated testing. I introduced them to Cruisecontrol and Ant for creating a CI build environment, the head of development (my boss) at the time was very interested in implementing a unit testing regime for the newly migrated .Net code. I remember working with an intern who had been tasked with creating a set of guidelines for unit testing within the integration code, this was about the time I started looking at Cruisecontrol.Net and Nant.

Given that my new place was a .Net house, it made little sense to me to implement a suite of Java based tools when there were viable .Net versions available. With this said and done, the company branched out into the new realms of .Net CI! It was a success from the get go, the upper management liked the idea of seeing nice graphs and reports on build status, the developers like the idea of being told when a specific build passed testing - it allowed pretty much anyone in the developer and analyst pool to see what was happening with their build and others.

I feel that good CI is often based on some simple foundations. First off would have to be good code. It doesn't matter if it doesn't pass the build at a lower level, those guys should be used to new build to go through some teething problems with production code, this is something that CI would come to reduce further down the line. Next, we needed a decent environment in which to build our code. At the start, we used Crusiecontrol.Net on one builds server, but we had another three servers available to us should we need them. As unit testing was rolled out for all of development, this allowed everyone to see problems before a build made it out onto a test environment. Naturally, this annoyed some developers who didn't see the point in unit testing their code, being in the build development team I could see the importance of unit testing and so could my colleagues. The next important thing that makes up a good CI environment is Software Configuration Management. I have worked with some excellent SCM's and some truly awful (read Visual Source Safe). At this point the company was using a tool called Surround (made by Seapine). It was a fairly good SCM, a little clunky and had some performance issues, we needed to move away from it however as we needed to get to a point where we could run multiple projects from a common library of assemblies. Surround wasn't really up to this. Being a .Net house, we had access to all of the latest tools from MS, naturally Visual Studio was in there. At around about this point in time, MS was starting to promote a tool called Team Foundation Server, this was essentially MS's take on CI. It came with all all the bells and whistles that a good CI system should offer, there was built in SCM, built in reporting, built in data collection and the ability to carry out Team Builds (MS's take on a Cruisecontrol style too). It also came with a tool called MSBUILD, which I like very much. MSBUILD was the replacement for NMake, I am going to assume that most people reading this article will be familiar with Apache Ant and/or Nant. MSBUILD borrows a lot from these tools. Essentially, it is a XML based script that allowed you to automate a great deal of pre-build, build and post-build activities. At it's simplest, it allows you to carry out a build on the command line.

Now, when it came to the CI system that I built, I didn't want to use MSBUILD, at the time it seemed to constricted for me, instead I turned to what is probably MSBUILD's biggest rival, Nant. At the present time, both these tools are pretty much identical in what they can do and how they are built and maintained, back in the day though Nant offered slightly more flexibility over MSBUILD.

Now it came down to the system that I built. One of the strong points of Nant (and MSBUILD) is that you can write your own tasks for your build/test configuration. This is great when you need to add some small things into the config - this could be something as simple as plugging in a third party tool etc. Both of these tools offer a lot of functionality from the get go, all I had to write for the first CI system there was integration with our SCM, I needed to create some Nant tasks that would work with our SCM.

When we first went live with this system, there were some complaints. These mainly came from the developers working on new build, they would now be informed of breaks very early on in the build process automatically, Cruisecontrol.Net allowed you to do this. There was some resistance to unit testing also at this time, some developers simply did not want to see their code fail at such an early stage. Both the head of development and myself held the same view - unit tests are a must have as they find issues way in advance of a build making it out into test, where an analyst may or may not find the issue. Also, the build development team was also subject to unit tests, so across the entire company unit testing became a must be done (especially when managerial start looking at lovely reports that showed them test coverage and pass rates. A lot of the developers that complained were of the "just get it to run" frame of mind. Thankfully, these developers were vastly in the majority.

As time progressed, we found that Cruisecontrol.Net was beginning to be a little constrictive for us. We were only using it for build and test for the entire company (read multiple projects) on one build server. By this time, upper management was extremely impressed with what had been achieved with the setup, it has sped up the build and test time across the board. When companies are successful, they often start branching out to get more clients to use their systems, this was no different to the one I was currently working for. It was financial software, so the money to be made from efficient development was astronomic. As the shape of the overall development changed, so did our attitude to building it changed also. With the current setup, we couldn't handle these sorts of builds on a daily basis, at this point we were running CI for one workstream only and the system would not be able to cater for different builds for different projects. We were under using the build environment we had, we only used one machine out of the four available to us. We knew we needed to change our build tools and management was concerned that we would loose development time having to migrate from one system to another. We quickly put that fear down by explaining that the build scripts we were using were not dependant on the CI system, in other words, the Nant scripts we used for all of the builds would work with any CI tool. This brought a sigh of relief to many a manager a team lead. Right up until this point, we were running a complete build and test for the new build for just one customer automatically. This would take place all the way through the day at scheduled times, this meant that developers could work away to their hearts content pretty much 24/7 (and some did come close to this).

This made moving to a new CI system fairly painless, it also allowed us to see what we could do for the build, not just a test and build but hopefully a test, build, deployment and further testing. A complete lights off system that would roll back to a working build should the current code fail.

Building a data access layer

2009-11-03T06:43:00.000-08:00

I recently agreed to help a student out with a software development assignment out for university. At first glance, it looks to be a pretty heavyweight peice of work for an undergrad to be given, even more so when the undergrad doesnt have a lot of experience in software development.

Essentially, the assignment is to create an Ebay-esque Windows Form app. The wording of the assignment makes it look harder than it actually is, when you get around the features that are being requested the actual technical requirement is really very simple. The assignment needs to be split into several peices, something that a lot of students probably wont be familiar with as it simply doesnt get taught at uni. The first section is the persistent data storage, I don't yet know what options are available at the moment, but the assignment spec asks for data to be retrieved from either a database or some XML. Now, I can understand why these features would be requested, it allows the application to be able to pick up its data from the local infrastructure or from a web service.

So, they key technical part here? Well, it's a data access layer. I have put together a few of these now, normally as part of an architectural migration to ease the switch over, all this assignment really needs is an appropriate DAL in order to get its data into the Windows app. It struck me that there really isnt a lot of info on how to do this rather simple thing, I think with the maturation of ORM like nHibernate and Linq-to-SQL a lot of people are over looking this for smaller projects. Anyway, I plan to put together a short howto on building your own DAL, because once you understand how the things work in the firs place, it makes implementing your own or using one of the ORM's already out there a lot more intuitive.

Old School Computing, in its most truest sense.

2009-10-11T05:58:00.000-07:00

Recently, I posted about the excellent machines that a rejuvinated Acorn Computers are now selling on the Internet. It got me thinking, anyone who went through an English school suring the late 80's and early 90's should have at least some distant memory of these rather fine machines.

For those that don't know, Acorn was one of the UK's only computer builders, a lot of people wont even remember them these days, but the ones who do will no doubt have fond memories of the BBC Master and probably playing Chuckie Egg on it.

Acorn made computers from 1978 to 1998, when they finally succumbed to the dominance of the PC architeture. There is loads about them on Wikipedia: http://en.wikipedia.org/wiki/Acorn_Computers, there are loads of these machines about now, mainly because of the sheer amount that were installed in schools across the country for so long. The first ever computer I owned was an Acorn Electron back in the early 80's. My parents thought that it might improve my hand/eye coordination to play games on it!! I think what many people wont realise is the lasting and prominent impact Acorn Computers have had on our everyday lives. Not only did they build their own computers, but they made their own OS, architecture and hardware. Acorn are resposible for developing the ARM processor, a peice of hardware that pretty much everyone carries around in their pocket, powering their mobile phone. It would be a fair bet to make that there is also one in your set top box at home, powering Sky, Virgin Media or your freeview/sat tuner. It doesnt start and end there either, ARM processors powered another venerable computing device towards the end of the 90's, the PSION Series 5. This is a very notable device as it was the precursor to the S60 mobile phone platform, the OS that powered it was called EPOC and it pretty much turned into Symbian - now things should start to be recogniseable!

With that said, I have now in front of me an original Acorn A7000+ desktop computer from a school down in England. It should run, I have powered it on and got the desktop up and displayed on my monitor. I plan to rennovate this UK computing relic to get it back up and running properly. I remember schools were using these machines right up until 2001 in some parts of the UK, this one I have here is a clear indication as it comes with an all important ethernet adapter.

One of the notable things about these machines is that they don't really need hard drives, the entire OS and all its supporting applications are stored on ROM. When you want to upgrade the OS, you simply replace the ROMs with your new set. This isn't something we see at all these days in a desktop computer, unless you are running a setup with embedded Linux.

I recall from the last time I saw an Acorn machine in operation, that it had a fully functional web browser. So, my plan right now is to get this bad boy up and running and connected to the Internet so that I can try and do some simple things like email and facebook. The example I have here, an A7000+, has a hard drive installed, as well as a CD writer, so this means I will be able to install software to it if I need to. From what I can see from the boot up screen, this machine is running RISC OS 3.7. From what I can remember, this was the last main release of the OS prior to Acorns demise, however since then there have been three seperate releases of newer OS's. In a rather different move to the normal distributions of Windows and Linux etc that we see now, a new company was formed in 1999 to release RISC OS 4. Not long after this, a seperate company release a RISC OS 5 for a seperate line of devices, but as far as I can tell this version of the OS is only for machines made by a company called Castle. This confuses me a bit, as far as I can tell this version of the OS has never really been available seperately.

But, a further twist is this, a new OS has been recently released, RISC OS 6... I have no idea how much of a departure this will be from the RISC OS I know and love. One thing that is very notable about RISC OS is that it pretty much introduced the concept of the task bar (as far as I know). This is very familiar to Apple users and now, to Windows 7 users, its a feature that Acorn introduced in 1988 though... As soon as I get this one up and running, I will start to get some screen shots up and running.

Sorting Lists with Comparison in .Net

2009-08-07T11:51:00.000-07:00

The other day, I made a post about invoking methods using reflection. One of the things I wanted to do in my code was get a list of the methods in a class and then run them. However, what I didnt mention was that I wanted the methods added to my list in an order. Sorting a list is easy if you are working with a list of strings or numbers. But the list I generated was a list of type MethodInfo - something that .Sort wouldnt work with.

So, to overcome this, I did the following:

sectionNames.Sort(delegate(MethodInfo m1, MethodInfo m2) { return m1.Name.CompareTo(m2.Name); });

And out pops a sorted list of type MethodInfo. Short, but sweet and also provides a nice little example of how lists of different types can be sorted.

Sick of my phone

2009-07-31T15:39:00.000-07:00

I am totally sick of my phone just now. For the past five years I have been using Nokia phones, mainly S60 smartphones. However, over the last few years I have been getting more and more frustrated with the S60 OS. It sees to me that once Nokia gained control of Symbian (at least I think it did) things started getting bad.

At the present time, I use an N96 - when it wants to work. Recently, one of my friends at work got hold of the HTC Hero, a smartphone that uses Google's Android 1.5. As far as I am concerened, it out cools the iPhone. However, I dont know if I want to get another smartphone. Right now I am thinking about getting a normal phone and some form of Internet Tablet/PDA - ironically I am looking at getting a Nokia N810. My reasons for this? Well, out of the box it runs a Linux, but you can also install Android on it as well :).

PDF Generation with PDFJet

2009-07-31T15:31:00.000-07:00

As mentioned previously, I have been working a lot on report generation. Where I work just now, we produce a lot of reports. A lot of them are produced manually and are very large. An ideal candidate for automation.

I need to produce them in a number of different formats, pdf is the first one I tackled. All of the reports I need to produce use tables, so I needed to find a .Net pdf library that had this functionality. I took a look at what was on offer, open source, closed source and commercial. I went for a lib called PDFJet. This is a fairly mature Java pdf library that has been ported over to .Net for C# http://www.pdfjet.com/. Its fairly cheap and whilst it lacks a lot of in depth examples, there are quite a few available on their website. The support is really good too - if you have a question, the guys are able to provide feedback very quickly.

It is very well featured, but I am basically using it to generate tables from datatables. I have been using it in earnest for about a week now, so I will try and post some examples of how it can be used here. There are two ways of creating a table with PDFJet, you can use a delimited text file holding the data you need or you can build a table programatically. Doing it in the code is quite tricky, to do this you need to build a list of lists of type cell - not as easy as it sounds... However, I am very impressed with this library and the things it can do.

Automatically invoke methods from a given class

2009-07-31T14:58:00.000-07:00

Had an interesting problem at work this week. I have been working on a way to automatically generate a number of (lengthy) reports in a number of different formats. Each report has a number of sections that in turn hold a number of subsections. Each subsection holds either a chart or a table. Each report needs to be presented as either a pdf, a doc, an xls or a csv file. Some of the sections in each report are used in other reports.

I came round to designing each section as a class, in each class is a method that generates either a datatable to build a table from or a chart generated as an image. I wanted the generation of these datatables and images to be as generic as possible, the methods called provide the objects I needed but the way I call them needed to be as generic as I could get it. So, instead of putting together a method that calls all of these other methods I came up with the idea of building a list of all the methods I need to call in order to generate a report. To do this, I used reflection.

Firstly, I needed to get a list of all the method names I wanted to call.

List<MethodInfo> sectionNames = new List<MethodInfo>();
Sections s = new Sections(conn);
Type t = s.GetType();
MethodInfo[] mi = t.GetMethods();
foreach (MethodInfo method in mi)
{
sectionNames.Add(method);
}

In this example, Sections is the name of the class holding the methods I need to generate a report. It populates its datatables from a database, so the conn variable holds the connection string needed. There are a lot of sections in the report I was working on, so I split each section up as partial classes. Using GetMethods () on my class returns all of the available methods within that class. With that information, I simply create a list of type MethodInfo and add each element of the array I have created to the list. I dont really need to do this, but at some point I am going to want to sort the list, which is a little easier than sorting the array (from my perspective). I now have a list of all the method names available in my class. The next thing I do is get rid of some of the generic methods, like GetType(). I dont know if there is a cleaner way of doing this, so all I am doing right now is deleting a range from the list:

sectionNames.RemoveRange(8, 4);

Now I have almost everything I need. Next up, I need to invoke the method and capture its return. The first type of report I tackled was a pdf. The library I am using to create these pdf's uses delimited text files to create tables (it can be done programatically, but I havent mastered this yet). So, I need to create a dictionary holding the name of the text file I want to create and the name of the method to create it from. Now, I also mentioned that my report will also hold charts generated as images - so the dictionary I create needs to hold a string and an object, not a string and a datatable or any other type. It may not be the cleanest way to do it, but I can cast the object as a datatable later on, anyway...

Dictionary<string, Object> content = new Dictionary<string, Object>();

Thats my dictionary ready to go, I just need to populate it with filenames and object. This isnt as hard as it seems, because my list contains type MethodInfo I can use the Invoke method to call all of the methods in the list. Thats a bit of a mouthful really, so it is better to demonstrate:

foreach (MethodInfo method in sectionNames)
{
object obj = method.Invoke(s, null);
content.Add(method.Name + ".txt", obj);
}

I loop through each MethodInfo type in my list. I create an object called obj, this becomes the return from each of the methods I call using Invoke. Once called and invoked, I add the result to my dictionary, on success the dictionary contains a key value pair representing a string filename and an object. You can see in the example above that the filename is always a .txt - this is bad when it comes to images (I am not creating any just yet). But, this can be worked around by getting the type of the object etc. However, right now this is all I need to create my text files to use with my pdf generation. I simply loop through the dictionary creating a textfile from the contents of the object (which is a datatable dont forget), the string is the filename the text file is created with.

Its not very efficient just now, I need to manage the methods in the dictionary - I need to remove the generic ones prior to adding them and I need to sort them in order, both things I either do manually or not at all right now. But this is a nice little example of how methods can be gathered from a class and then dynamically invoked. The complete source for this example is as follows:

/// <summary>
/// Creates a Dictionary of strings and object representing the content we want to build
/// </summary>
/// <returns>a Dictionary of strings and objects holding the filename and the datatable</returns>
private Dictionary<string, Object> Contents()
{
List<MethodInfo> sectionNames = new List<MethodInfo>();
Sections s = new Sections(conn);
Type t = s.GetType();
MethodInfo[] mi = t.GetMethods();
foreach (MethodInfo method in mi)
{
sectionNames.Add(method);
}
sectionNames.RemoveRange(8, 4);
sectionNames.Sort();
Dictionary<string, Object> content = new Dictionary<string, Object>();
foreach (MethodInfo method in sectionNames)
{
object obj = method.Invoke(s, null);
Console.WriteLine(obj);
content.Add(method.Name + ".txt", obj);
}
return content;

I will also post the file creation method that works with this after the weekend as well as some bits and peices to do with the pdf library I am using. But hopefully this should be enough to get someone else on the right track if they are also considering such a soloution.

Old School Acorn Computing

2009-06-21T05:03:00.000-07:00

Just been doing a bit of a browse for info on my vintage Acorn A7000+ desktop when I discovered that Acorn Computers are back in business. I was impressed, however a little dismayed to see that they dont use the classic RISC OS operating system any more, but instead plump for Vista, or other versions of Windows. Their site is here: http://www.acorncomputers.co.uk/index.htm, and to be honest the devices they are selling do seem to be quite attractive. They come complete with the old school Acorn logo all over them, which just adds to the appeal.

I think anyone who still has fond memories of using Acorn and BBC computers when they were at school will enjoy taking a look at the new series of products. Of course, anyone who is still hankering for a bit of RISC OS action can take a look at Ebay, you can pick up an A7000+ very cheaply these days, some even with a network adapter. Schools are literally chucking these things out with the rubbish nowadays, unfortunately they can't even give them way to charities etc. However, if you really want the latest iteration of RISC OS, currently at version 5, check out Castle's list of products here: http://www.iyonix.com/. Personally, I dont see the need to spend almost £800 on a computer like this, but if you really are an enthusiast you will snap one right up.

PGP Decryption with C#

2009-06-07T08:19:00.000-07:00

Yesterday I posted on how I was able to encrypt some data and compress it using C#. Been thinking about how useful it would be to be able to decrypt it as well, there is no pressing need for me to do this in the situation I was working in really - once my data is encrypted and archived it would be sent off to someone else, so I dont really need to worry about decrypting it.

Thing is, I just wondered how it would work. I already know that Bouncy Castle works heavily with streams, so I guess the logic would be:

Open a file as a stream

Get the encryption keys needed to decrypt the file

Decrypt the file to a stream

Write the file(s)

I already have the code that will manage the keys for me and I already have an understanding on how Bouncy Castle's libs work with streams. I had a quick google to see if there was anything going in the way of examples on how to do this, but only managed to find one (someone had pasted the example in the Bouncy Castle source to a forum). Thing is, none of the offered examples seemed to fit my code, which was a shame. However, these things can be overcome once you understand a little more about how Bouncy Castle implements OpenPGP. The code I created to decrypt my archive isnt going to go into production (probably), its just something I wanted to try out, so it's all one big method. Note, these libs work on streams - to decrypt my archive and get to the literal data, I had to manipulate the streams to get there, I think this could put a lot of people off using this. First off, we need to get the file we wish to decrypt into a stream and determine the decoder from it:

input = PgpUtilities.GetDecoderStream(input);

The next thing we want to do is create a PGPObjectFactory. From here we can get a PGPEncryptedDataList and start creating objects from our input stream:

PgpObjectFactory pgpObjF = new PgpObjectFactory(input);
                PgpEncryptedDataList enc;
                PgpObject obj = pgpObjF.NextPgpObject();
                if (obj is PgpEncryptedDataList)
                {
                    enc = (PgpEncryptedDataList)obj;
                }
                else
                {
                    enc = (PgpEncryptedDataList)pgpObjF.NextPgpObject();
                }

Now we are able to get the first encrypted object from our stream so lets decrypt the stream using the private key:

PgpPrivateKey privKey = pgpKeys.PGPPrivateKey;
                PgpPublicKeyEncryptedData pbe = null;
                foreach (PgpPublicKeyEncryptedData pked in enc.GetEncryptedDataObjects())
                {
                    if (privKey != null)
                    {
                        pbe = pked;
                        break;
                    }
                }
                Stream clear = pbe.GetDataStream(privKey);
                PgpObjectFactory plainFact = new PgpObjectFactory(clear);
                PgpObject message = plainFact.NextPgpObject();

With this done, we are now able to start inspecting the decrypted objects we are creating. We know the file we made earlier was an encrypted zip file, so at this point it is fair to assume message is compressed, which it is:

if (message is PgpCompressedData)
                {
                    PgpCompressedData cData = (PgpCompressedData)message;
                    Stream compDataIn = cData.GetDataStream();

I knew that message was of the type PGPCompressedData from inspecting the object, what we need to do now is find out when message becomes literal data. To do this, we need to go a bit further, I need to create a new PGPObjectFactory:

PgpObjectFactory o = new PgpObjectFactory(compDataIn);
                    message = o.NextPgpObject();
                    if (message is PgpOnePassSignatureList)
                    {
                        message = o.NextPgpObject();
                        PgpLiteralData Ld = null;
                        Ld = (PgpLiteralData)message;
                        Stream output = File.Create(outputpath + "\\" + Ld.FileName);
                        Stream unc = Ld.GetInputStream();
                        Streams.PipeAll(unc, output);
                    }

I had thought that the literal data would come next, but it didnt - dont forget when encrypting the data previously I signed it - so I got the signature first. So, I just needed to move on to the next object which was the literal data. As you can see above, all I need to do now is write the file out to the disk and I am done. I did think when I had completed this, "What about decompressing directory structures?". Well, in my humble opinion, there is no need to do this. From what I have seen with encryption over the last few months, files get encrypted, not directories. I wrote a lib the other day that compresses into a zip, I will post it here over the next week, it also decompresses (including the directory structure). So the way I see it, if you wanted to encrypt a set of files and directories, you would create the zip first and then create an encrypted archive from that file. I know it would be two seperate operations, but this sort of suits my situation at the moment: I dont need to decrypt archives programatically, but I can if I need to (now, anyway). So with the stuff I have now, I can create a zip for internal deployment and then create a seperate, encrypted zip from that file for external deployment.

The complete code for decryption is as follows:

using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using Tools.PGP.Crypto;

namespace Tools.PGP.Crypto
{
    public class PGPDecrypt
    {
        public string _encryptedFilePath;
        public string _privKeyPath;
        public char[] _password;
        public string _outputPath;
        public PGPKeys pgpKeys;
        public PGPDecrypt(string encryptedFilePath, string privKeyPath, string password, string outputPath, string pubKeyPath, long keyID)
        {
            _encryptedFilePath = encryptedFilePath;
            _outputPath = outputPath;
            _password = password.ToCharArray();
            _privKeyPath = privKeyPath;
            pgpKeys = new PGPKeys(pubKeyPath, privKeyPath, password, keyID);
        }
        public void decrypt(Stream input, string outputpath)
        {
            input = PgpUtilities.GetDecoderStream(input);
            try
            {
                PgpObjectFactory pgpObjF = new PgpObjectFactory(input);
                PgpEncryptedDataList enc;
                PgpObject obj = pgpObjF.NextPgpObject();
                if (obj is PgpEncryptedDataList)
                {
                    enc = (PgpEncryptedDataList)obj;
                }
                else
                {
                    enc = (PgpEncryptedDataList)pgpObjF.NextPgpObject();
                }
                PgpPrivateKey privKey = pgpKeys.PGPPrivateKey;
                PgpPublicKeyEncryptedData pbe = null;
                foreach (PgpPublicKeyEncryptedData pked in enc.GetEncryptedDataObjects())
                {
                    if (privKey != null)
                    {
                        pbe = pked;
                        break;
                    }
                }
                Stream clear = pbe.GetDataStream(privKey);
                PgpObjectFactory plainFact = new PgpObjectFactory(clear);
                PgpObject message = plainFact.NextPgpObject();
                if (message is PgpCompressedData)
                {
                    PgpCompressedData cData = (PgpCompressedData)message;
                    Stream compDataIn = cData.GetDataStream();
                    PgpObjectFactory o = new PgpObjectFactory(compDataIn);
                    message = o.NextPgpObject();
                    if (message is PgpOnePassSignatureList)
                    {
                        message = o.NextPgpObject();
                        PgpLiteralData Ld = null;
                        Ld = (PgpLiteralData)message;
                        Stream output = File.Create(outputpath + "\\" + Ld.FileName);
                        Stream unc = Ld.GetInputStream();
                        Streams.PipeAll(unc, output);
                    }
                    else
                    {
                        PgpLiteralData Ld = null;
                        Ld = (PgpLiteralData)message;
                        Stream output = File.Create(outputpath + "\\" + Ld.FileName);
                        Stream unc = Ld.GetInputStream();
                        Streams.PipeAll(unc, output);
                    }
                }
            }
            catch (Exception e)
            {
                throw new Exception(e.Message);
            }
        }
    }
}

To decrypt a file, I can just call the method like so:

PGPDecrypt test = new PGPDecrypt(@"C:\test\somefile.zip",
                                             @"C:\GnuPG\secring.gpg",
                                             "password",
                                             @"C:\test\test",
                                             @"C:\GnuPG\pubring.gpg",
                                             3699527550217851901);
            FileStream fs = File.Open(@"C:\test\somefile.zip", FileMode.Open);
            test.decrypt(fs,@"C:\test\test");

PGP Encryption with C#

2009-06-06T02:53:00.000-07:00

Recently at work, I had to come up with a deployment framework for a legacy app we use. The initial requirement was to simply copy the app from a network share onto users desktops. There was also a requirement to provide a level of security around the deployment itself, so to me, an anonymous network share didnt seem to be the correct way of releasing this app... A much more simpler way to provide at least a small level of security would be make the deployment framework download the app as an archive from a webserver on the domain. This, at least, means that only users with authorisation to access the domain would be able to download the app, and they would only be able to download it whilst they were logged in.

Then I was told that there will be some instances that field workers will need to download parts of the app whilst they are out and about doing whatever it is they do. I had already planned to compress the app as an archive, but now I needed a way to secure the archive itself. One of my colleagues suggested that I put a password on the archive, its not a daft suggestion to make, it would imply that only people with the password would be able to extract the data from the archive itself. But, I pointed out that this doesnt prove that the archive was one created by us - all it does it password protect an archive. Plus, we would need to communicate the password to those that needed it in a secure manner, which in turn would mean we couldnt use the same password all the time etc. I asked if we already use some form of secure signature and got told we use PGP - result. I explained to my colleagues that yes, password protecting the archive would mean that only password holders could extract from the archive, but this wouldnt mean they are authorised to hold the password in the first place. I also explained that the password could be cracked, even if it was sufficiently complex, a persistant attacker could spend a lot of time trying to crack it.

With this in mind, I went on to say that as we already encrypt data with PGP, we could do the same thing here with the archives we are planning to create. This way, people using our app (or archives in general) would be able to see with great certainty that they did originate from us and that it hasnt been substituted by a third party.

This is part of a .Net project, so any encryption would ideally take place using the .Net framework. However, right now there is no PGP support in .Net. There is support for cryptograhy in general, for instance RSA is employed as well as DPAPI, but we already use PGP and already have public keys. So it would make more sense to me for us to use these for encryption. To do this, I am going to be using the C# cryptography libraries available from The Legion of the Bouncy Castle: http://www.bouncycastle.org/csharp/ . There are not a lot of good examples of how to use this out on the Internet, the best I could find to outline the concepts involved was here. Unfortunately, the formatting of the page makes it a little hard to understand what is happening - which is the important part

Lets get down to encrypting a file and compressing it to an archive. I am going to assume that if you are searching for PGP encryption methods in C# that you have already generated a set of PGP keys. If you havent, then I can suggest that you nip over to http://www.gnupg.org/ and download a copy of GnuPG to generate some for you (it's what I used to test through my app). The first step we need to take is to understand what we need to get the key values. We need to know the path to the public and private keys, the password for the public key to encrypt with and we need to know the key ID we want to target. The key ID is a 64bit signed integer, using GnuPG you can get this value from the command line using the command:

--list-keys --with-colons <userid>

Simply replace the user ID with the user ID attached to the key (this will probably be the email address used when generating the key itself). You will get something that looks like this back:

pub:u:1024:1:33575CB4BB8193FD:2009-06-06:::u:Your Name(RSA Sign only test) <your@email.address>::scESC:
sub:u:1024:1:31B23C6C6681F46B:2009-06-06::::::e:

The key ID is the bit in bold (or should be bold, but not quite so well - it's the fith column along). This is the key ID in hexidecimal, in the code I am about to show you it needs to be in decimal format, so you can either convert it in c# with the following:

int decAgain = int.Parse(hexValue, System.Globalization.NumberStyles.HexNumber);

Or you can use a tool to convert it for you like this one: http://www.parkenet.com/apl/HexDecConverter.html, or if you happen to know what it is anyway (like I did), you can just type it in etc.

I used RSA encryption when generating my key. I used this as it would appear to be the best method for creating keys right now, there are not many examples of how to use PGP encryption on the Internet at the moment, and the peices of code that are here and there seem to use older standards.

Right, now we know our key ID etc, we can start writing the code. I created a class called PGPKeys to do all my key related work for me. First up, we need to declare all our variables etc:

/// <summary>
        /// long representing the key ID we want to target.
        /// </summary>
        private long _keyID;
        /// <summary>
        /// string to represent the path to the private key file.
        /// </summary>
        private string _privKeyPath;
        /// <summary>
        /// string to represent the path to the public key file.
        /// </summary>
        private string _pubKeyPath;
        /// <summary>
        /// string to represent the passphrase used with PGP.
        /// </summary>
        private string _password;
        public PgpPublicKey PGPPublicKey { get; private set; }
        public PgpPrivateKey PGPPrivateKey { get; private set; }
        public PgpSecretKey PGPSecretKey { get; private set; }

Nice and simple. Next up, I want to be able to validate the values I am planning to use:

/// <summary>
        /// Constructor to manage variables for us.
        /// </summary>
        /// <param name="pubKeyPath">string representing the public key path</param>
        /// <param name="privKeyPath">string representing the private key path</param>
        /// <param name="password">string representing the passphrase</param>
        /// <param name="keyID">long representing the key ID we want to use</param>
        public PGPKeys(string pubKeyPath, string privKeyPath, string password, long keyID)
        {
            if (!File.Exists(pubKeyPath))
            {
                throw new ArgumentNullException("Could not find the public key at " + pubKeyPath);
            }
            else
            {
                _pubKeyPath = pubKeyPath;
            }
            if (!File.Exists(privKeyPath))
            {
                throw new ArgumentNullException("Could not find the private key at " + privKeyPath);
            }
            else
            {
                _privKeyPath = privKeyPath;
            }
            if (String.IsNullOrEmpty(password))
            {
                throw new ArgumentNullException("The password must not be null");
            }
            else
            {
                _password = password;
            }
            if (keyID == 0)
            {
                throw new ArgumentNullException("The key ID must not be null");
            }
            else
            {
                _keyID = keyID;
            }
            PGPPublicKey = getPublicKey(_pubKeyPath);
            PGPSecretKey = getSecretKey(_privKeyPath);
            PGPPrivateKey = getPrivateKey(_password);
        }

You can see here where the public, private and secret keys are going to be held. Simple so far, but now the work begins. I think one of the tough things to get your head around when using the Bouncy Castle libs is that everything is a set of streams that have things done to them. This is a slight over simplification, but the following bits of code will demonstrate this process. Lets look at getting the public key value:

/// <summary>
        /// private method to get the public key value.
        /// </summary>
        /// <param name="_pubKeyPath">string representing the public key path</param>
        /// <returns>a PGPPublicKey</returns>
        private PgpPublicKey getPublicKey(string _pubKeyPath)
        {
            PgpPublicKey pubKey;
            using (Stream keyin = File.OpenRead(_pubKeyPath))
            using (Stream s = PgpUtilities.GetDecoderStream(keyin))
            {
                PgpPublicKeyRingBundle pubKeyBundle = new PgpPublicKeyRingBundle(s);
                pubKey = pubKeyBundle.GetPublicKey(_keyID);
                if (pubKey == null)
                    throw new Exception("The public key value is null!");
            }
            return pubKey;
        }

This is the on ramp with PGP, getting the public key value represented in the code itself. It also shows how Bouncy Castle likes to work with streams, the first stream here opens the file in which the key is stored, the second stream then goes on to get the value from it. With that done, lets get the secret key:

/// <summary>
        /// private method to get the secret key value.
        /// </summary>
        /// <param name="_privKeyPath">string representing the private key path</param>
        /// <returns>a PGPSecretKey</returns>
        private PgpSecretKey getSecretKey(string _privKeyPath)
        {
            PgpSecretKey secKey;
            using (Stream keyin = File.OpenRead(_privKeyPath))
            using (Stream s = PgpUtilities.GetDecoderStream(keyin))
            {
                PgpSecretKeyRingBundle secKeyBundle = new PgpSecretKeyRingBundle(s);
                secKey = secKeyBundle.GetSecretKey(_keyID);
                if (secKey == null)
                    throw new Exception("The secret key value is null!");
            }
            return secKey;
        }

Here we pretty much do the same thing as before, but this time we are looking at the private key file path. Dont get confused here, this method gets the secret key value, to do this we point it at the private key file and supply a key ID. To get the private key itself, we use the following:

/// <summary>
        /// private method to get the private key value.
        /// </summary>
        /// <returns>a PGPPrivateKey</returns>
        private PgpPrivateKey getPrivateKey(string _password)
        {
            PgpPrivateKey privKey = PGPSecretKey.ExtractPrivateKey(_password.ToCharArray());
            if (privKey == null)
            {
                return null;
            }
            else
            {
                return privKey;
            }
        }

This one is a lot smaller, but with it we get our private key. And with that, we are done with our PGPKeys class, lets move on to encrypting, signing and compressing using these keys. Remember what I said about streams? Well, this is how it comes into play, lets encrypt some data:

private Stream encrypt(Stream output)
        {
            PgpEncryptedDataGenerator pgpEncDataGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Twofish, new SecureRandom());
            pgpEncDataGen.AddMethod(_pgpKeys.PGPPublicKey);
            Stream encryptedOutput = pgpEncDataGen.Open(output, new byte[BufferSize]);
            return encryptedOutput;
        }

Thats it done, this will encrypt a stream only. On its own, this isnt so useful, you can see where I am specifying the encryption algorythm (I decided to use TwoFish, apparently this is a very secure algorythm) and where the public key value is applied. If you just hold in your mind for a moment that this represents a stream of encrypted data at its conclusion, we can look at how it is compressed to an archive:

private Stream compress(Stream output)
        {
            PgpCompressedDataGenerator pgpCompDataGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
            Stream compressedEncryptedOut = pgpCompDataGen.Open(output);
            return compressedEncryptedOut;
        }

The stream we just emcrypted is now passed to this method which compresses it to a zip - right now though, this is still a stream, nothing more. The next thing we need to do is apply the literal data to this stream:

private Stream literalOutput(Stream compressedOut, FileInfo file)
        {
            PgpLiteralDataGenerator pgpLiteralDataGen = new PgpLiteralDataGenerator();
            Stream literal = pgpLiteralDataGen.Open(compressedOut, PgpLiteralData.Binary, file);
            return literal;
        }

Now we have a stream that has been encrypted, compressed and had its literal data applied to it. I am going to point out here that if you dont want to create a PGP encrypted file, but instead just want a PGP encrypted stream, then you could just use the encrypt method. Anyway, now we are ready to write and sign our archive:

private static void writeAndSign(Stream ouput, Stream literalout, FileStream inputFile, PgpSignatureGenerator sigGen)
        {
            int length = 0;
            byte[] buf = new byte[BufferSize];
            while ((length = inputFile.Read(buf, 0, buf.Length)) > 0)
            {
                literalout.Write(buf, 0, length);
                sigGen.Update(buf, 0, length);
            }
            sigGen.Generate().Encode(ouput);
        }

The last thing we want to do here, is sign the encrypted data itself:

private PgpSignatureGenerator sigGen(Stream compressedOut)
        {
            const bool Iscritical = false;
            const bool IsNested = false;
            PublicKeyAlgorithmTag tag = _pgpKeys.PGPSecretKey.PublicKey.Algorithm;
            PgpSignatureGenerator pgpSigGen = new PgpSignatureGenerator(tag, HashAlgorithmTag.Sha1);
            pgpSigGen.InitSign(PgpSignature.BinaryDocument, _pgpKeys.PGPPrivateKey);
            foreach (string userID in _pgpKeys.PGPSecretKey.PublicKey.GetUserIds())
            {
                PgpSignatureSubpacketGenerator subPackGen = new PgpSignatureSubpacketGenerator();
                subPackGen.SetSignerUserId(Iscritical, userID);
                pgpSigGen.SetHashedSubpackets(subPackGen.Generate());
                break;
            }
            pgpSigGen.GenerateOnePassVersion(IsNested).Encode(compressedOut);
            return pgpSigGen;
        }

Now we are able to encrypt, compress, sign and write our chosen file to an archive. All we need to do now is tie it up nicely:

public void EncryptSignAndZip(Stream output, FileInfo unencryptedinput)
        {
            if (output == null)
            {
                throw new ArgumentNullException("The output stream cannot be null");
            }
            if (unencryptedinput == null)
            {
                throw new ArgumentNullException("You must supply a filename to encrypt");
            }
            if (!File.Exists(unencryptedinput.FullName))
            {
                throw new ArgumentNullException(unencryptedinput + " does not exist");
            }
            using (Stream encryptedout = encrypt(output))
            using (Stream compressedOut = compress(encryptedout))
            {
                PgpSignatureGenerator signature = sigGen(compressedOut);
                using (Stream literalOut = literalOutput(compressedOut, unencryptedinput))
                using (FileStream inputfile = unencryptedinput.OpenRead())
                {
                    writeAndSign(compressedOut, literalOut, inputfile, signature);
                }
            }
        }

Again, I am validating the arguments as they go in to make sure that they are not null values. You can also see here how Bouncy Castle works with the streams we are creating. However, thats it - these classes will create an encrypted zip file for you with what ever you want in them. To make use of this class, I used this in testing:

string pubKeyPath = @"C:\GnuPG\pubring.gpg";
            string privKeyPath = @"C:\GnuPG\secring.gpg";
            string password = "password";
            long _keyID = 3699527550217851901;
            PGPEncrpyt test = new PGPEncrpyt(pubKeyPath, privKeyPath, password, _keyID);        
            FileInfo myFile = new FileInfo(@"C:\test\app.config");
            FileStream mystream = new FileStream(@"C:\test\somefile.zip", FileMode.Create, FileAccess.Write);
            test.EncryptSignAndZip(mystream, myFile);

And thats it, I using the code above I was able to create an encrypted zip, which I was able to decrypt and extract using GnuPG. I have written the code to decrypt it yet as at this time, I dont really need it - the files I am encrypting will normally be sent on to other organisations etc. One thing I may make use of in future is the ability to encrypt just a stream - this may become very useful for me when it comes to sending data over a network to other parts of the business. The complete code is as follows, PGPKeys:

using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace Tools.PGP.Crypto
{
    /// <summary>
    /// Nested class to get information on out PGP Keys
    /// </summary>
    public class PGPKeys
    {
        /// <summary>
        /// long representing the key ID we want to target.
        /// </summary>
        private long _keyID;
        /// <summary>
        /// string to represent the path to the private key file.
        /// </summary>
        private string _privKeyPath;
        /// <summary>
        /// string to represent the path to the public key file.
        /// </summary>
        private string _pubKeyPath;
        /// <summary>
        /// string to represent the passphrase used with PGP.
        /// </summary>
        private string _password;
        public PgpPublicKey PGPPublicKey { get; private set; }
        public PgpPrivateKey PGPPrivateKey { get; private set; }
        public PgpSecretKey PGPSecretKey { get; private set; }
        /// <summary>
        /// Constructor to manage variables for us.
        /// </summary>
        /// <param name="pubKeyPath">string representing the public key path</param>
        /// <param name="privKeyPath">string representing the private key path</param>
        /// <param name="password">string representing the passphrase</param>
        /// <param name="keyID">long representing the key ID we want to use</param>
        public PGPKeys(string pubKeyPath, string privKeyPath, string password, long keyID)
        {
            if (!File.Exists(pubKeyPath))
            {
                throw new ArgumentNullException("Could not find the public key at " + pubKeyPath);
            }
            else
            {
                _pubKeyPath = pubKeyPath;
            }
            if (!File.Exists(privKeyPath))
            {
                throw new ArgumentNullException("Could not find the private key at " + privKeyPath);
            }
            else
            {
                _privKeyPath = privKeyPath;
            }
            if (String.IsNullOrEmpty(password))
            {
                throw new ArgumentNullException("The password must not be null");
            }
            else
            {
                _password = password;
            }
            if (keyID == 0)
            {
                throw new ArgumentNullException("The key ID must not be null");
            }
            else
            {
                _keyID = keyID;
            }
            PGPPublicKey = getPublicKey(_pubKeyPath);
            PGPSecretKey = getSecretKey(_privKeyPath);
            PGPPrivateKey = getPrivateKey(_password);
        }
        /// <summary>
        /// private method to get the public key value.
        /// </summary>
        /// <param name="_pubKeyPath">string representing the public key path</param>
        /// <returns>a PGPPublicKey</returns>
        private PgpPublicKey getPublicKey(string _pubKeyPath)
        {
            PgpPublicKey pubKey;
            using (Stream keyin = File.OpenRead(_pubKeyPath))
            using (Stream s = PgpUtilities.GetDecoderStream(keyin))
            {
                PgpPublicKeyRingBundle pubKeyBundle = new PgpPublicKeyRingBundle(s);
                pubKey = pubKeyBundle.GetPublicKey(_keyID);
                if (pubKey == null)
                    throw new Exception("The public key value is null!");
            }
            return pubKey;
        }
        /// <summary>
        /// private method to get the secret key value.
        /// </summary>
        /// <param name="_privKeyPath">string representing the private key path</param>
        /// <returns>a PGPSecretKey</returns>
        private PgpSecretKey getSecretKey(string _privKeyPath)
        {
            PgpSecretKey secKey;
            using (Stream keyin = File.OpenRead(_privKeyPath))
            using (Stream s = PgpUtilities.GetDecoderStream(keyin))
            {
                PgpSecretKeyRingBundle secKeyBundle = new PgpSecretKeyRingBundle(s);
                secKey = secKeyBundle.GetSecretKey(_keyID);
                if (secKey == null)
                    throw new Exception("The secret key value is null!");
            }
            return secKey;
        }
        /// <summary>
        /// private method to get the private key value.
        /// </summary>
        /// <returns>a PGPPrivateKey</returns>
        private PgpPrivateKey getPrivateKey(string _password)
        {
            PgpPrivateKey privKey = PGPSecretKey.ExtractPrivateKey(_password.ToCharArray());
            if (privKey == null)
            {
                return null;
            }
            else
            {
                return privKey;
            }
        }
    }
}

And PGPEncrypt:

using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace Tools.PGP.Crypto
{
    /// <summary>
    /// Public class to encrypt and compress files to a zip archive using PGP
    /// </summary>
    public class PGPEncrpyt
    {
        /// <summary>
        /// instansiate a PGPKeys object
        /// </summary>
        private PGPKeys _pgpKeys;
        private const int BufferSize = 0x10000;
        public PGPEncrpyt(string _pubKeyPath, string _privKeyPath, string _password, long _keyID)
        {
            _pgpKeys = new PGPKeys(_pubKeyPath, _privKeyPath, _password, _keyID);
        }
        private static void writeAndSign(Stream ouput, Stream literalout, FileStream inputFile, PgpSignatureGenerator sigGen)
        {
            int length = 0;
            byte[] buf = new byte[BufferSize];
            while ((length = inputFile.Read(buf, 0, buf.Length)) > 0)
            {
                literalout.Write(buf, 0, length);
                sigGen.Update(buf, 0, length);
            }
            sigGen.Generate().Encode(ouput);
        }
        private Stream encrypt(Stream output)
        {
            PgpEncryptedDataGenerator pgpEncDataGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Twofish, new SecureRandom());
            pgpEncDataGen.AddMethod(_pgpKeys.PGPPublicKey);
            Stream encryptedOutput = pgpEncDataGen.Open(output, new byte[BufferSize]);
            return encryptedOutput;
        }
        private Stream compress(Stream output)
        {
            PgpCompressedDataGenerator pgpCompDataGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
            Stream compressedEncryptedOut = pgpCompDataGen.Open(output);
            return compressedEncryptedOut;
        }
        private Stream literalOutput(Stream compressedOut, FileInfo file)
        {
            PgpLiteralDataGenerator pgpLiteralDataGen = new PgpLiteralDataGenerator();
            Stream literal = pgpLiteralDataGen.Open(compressedOut, PgpLiteralData.Binary, file);
            return literal;
        }
        private PgpSignatureGenerator sigGen(Stream compressedOut)
        {
            const bool Iscritical = false;
            const bool IsNested = false;
            PublicKeyAlgorithmTag tag = _pgpKeys.PGPSecretKey.PublicKey.Algorithm;
            PgpSignatureGenerator pgpSigGen = new PgpSignatureGenerator(tag, HashAlgorithmTag.Sha1);
            pgpSigGen.InitSign(PgpSignature.BinaryDocument, _pgpKeys.PGPPrivateKey);
            foreach (string userID in _pgpKeys.PGPSecretKey.PublicKey.GetUserIds())
            {
                PgpSignatureSubpacketGenerator subPackGen = new PgpSignatureSubpacketGenerator();
                subPackGen.SetSignerUserId(Iscritical, userID);
                pgpSigGen.SetHashedSubpackets(subPackGen.Generate());
                break;
            }
            pgpSigGen.GenerateOnePassVersion(IsNested).Encode(compressedOut);
            return pgpSigGen;
        }
        public void EncryptSignAndZip(Stream output, FileInfo unencryptedinput)
        {
            if (output == null)
            {
                throw new ArgumentNullException("The output stream cannot be null");
            }
            if (unencryptedinput == null)
            {
                throw new ArgumentNullException("You must supply a filename to encrypt");
            }
            if (!File.Exists(unencryptedinput.FullName))
            {
                throw new ArgumentNullException(unencryptedinput + " does not exist");
            }
            using (Stream encryptedout = encrypt(output))
            using (Stream compressedOut = compress(encryptedout))
            {
                PgpSignatureGenerator signature = sigGen(compressedOut);
                using (Stream literalOut = literalOutput(compressedOut, unencryptedinput))
                using (FileStream inputfile = unencryptedinput.OpenRead())
                {
                    writeAndSign(compressedOut, literalOut, inputfile, signature);
                }
            }
        }
    }
}

Reporting made easy

2009-06-04T12:51:00.000-07:00

I recently found out at work that the MI analysts have a monthly report running up to seventy pages that they need to collate. Nothing unusual there, lots of places have big monthly reports. Then I was told that it takes five days to collate.

I initially though this was a joke, afterall this report had been publised each month for the last two years. Once I was told no, we really do do this, I needed to go and get myself a cup of sweet tea. After I had recovered, I started to speak to some of the analysts to try and find out the complete story behind this seemingly terrifying report, once I started to get some of the details behind this I came to the conclusion that the main reason for the length of time it takes to create this report was down to analysts doing the same thing every twenty-five days by hand. A prime opportunity to automate if ever there was one - and I had been after an excuse to create a .Net DAL for the team as well :).

There is nothing outstanding about this project at all, there isnt really any cutting edge tech being employed nor is there any new, fancy way of doing things being discussed. There is rarely anything taxing when it comes to generating reports. The most complicated thing you need to do is validate all the information that you are presenting, once the formula is there you are safe in the knowledge that your report will continue to churn out the required data for ever, the only time you will ever need to test it is when a new requirement is added to the schema of the report itself.

The uber report I am going to tackle just now is intended for publication via printed media. Because of this, I am confident in the knowledge that once the report has been created, no one will need to copy and past the data from it into another application - I know it gets distributed on paper. So, I know that I dont really need to provide a spreadsheet, csv, .doc or peice of XML initially. This is good news for me, as I only need to worry about one format, and I have decided to go for PDF. I had a look through the report a few days ago, when initially this task was given to me. Basically, its just a collection of graphs and tables, very little text, this just gets better as far as I am concerned. One format to worry about and very little wording to go into it - bliss!

So, what do I need for this to work? I need:

A graphing lib, I am going for ZedGraph. Its open source and does everything I need.

A PDF generation lob, for this I am going to use PFDJet. This is a commercial product, but there is also an open source version. As I dont need to do anything that fancy, it will do for me.

And thats pretty much it, all I will need to do is get the data I need from the db and use it to create some graphs and charts which go into a PDF report. Couldnt be simpler really, could it?

After a good planning meeting with the team lead and analyst I would be working with, it turns out that there is a similar system already in place for creating the basis for some of the data needed to create this report. After this session, I found that there is a monster Oracle DB somewhere in the depths of the core business that holds all the information we need. With the interesting bits to one side, I was then told a fairly comical story about Oracle, MySql and Access all doing a funny dance...

The only challenging thing really is the design of the report. Now that people have learned that I am automating some report generation, it seems like everyone has a preference as to how it should look. Right now, I am not concerned with fancy graphics or logos, right now the only important thing is getting the correct data into the report in a logical manner...

Encrypting Connections strings in .Net with NAnt

2009-05-23T00:37:00.000-07:00

I have been working a lot with databases over the last couple of weeks, one of the things that is a novelty to the people I am working with now is the ability to store multiple connection strings in our .config files. Its a really simple and flexible thing to do, allowing us to transport any of our apps from environment to environment with very little hassle. The only downer is that the connection strings are stored in plain-text, which naturally is a massive security hazard.

That is, of course, unless you encrypt them.

There are loads of examples on the Internet explaining how this can be done, and they are very simple to follow and implement - but I wonder how many people make use of this fairly simple feature? One thing that became apparent to me was, even though I can encrypt the file - just when and how do I want to do this? The when is simple, I want my connection strings encrypted when the application moves into a testing environment an I want them to stay encrypted from then on. The how is slightly different, there is no point in having the encryption take place as I am developing the app, as I want to have the flexibility of changing the connection strings as and when I need them. So, the ideal time for me to encrypt them would be during the build itself. Its a very straight forward thing to do, simply write a small program that encrypts the connection strings section of a web or app .config and have Visual Studio process this as a post build event. Its possibly the easiest way to do this. But, this isnt so flexible, it would only really encrypt one file relevant to a project, and whilst the code is reusable, it isnt very agile.

So, why not move this from being a post build event in the soloution file out to being a post build event in your larger build process? Using NAnt (or MSBUILD), we can create a custom task that will encrypt the connectionStrings section in any .config file. This could be used over and over again as and when requirements dictate a section in a config file should be encrypted.

There are a couple of things we need to capture:

The path of the .config file
What sort of encryption we want to use.

It would appear that .Net offers two methods to encrypt a .config file, DPAPI and RSA. DPAPI is provided by Microsoft, there is a nice article on Wikipedia discussing this encryption method. For a lot of people though, RSA may be a more suitable encryption method (its the one I am using if that is worth anything).

Back to the task. To start with, I want to get the location of the .config file and decide which encryption method I want to use. This is easy enough:

[TaskName("config-encrypt")]
    class Encryption : Task
    {
        private string strProvider;
        private string _filePath;
        [TaskAttribute("configfile", Required = true)]
        public string FilePath
        {
            get { return _filePath; }
            set { _filePath = value; }
        }
        private string _rsa;
        [TaskAttribute("RSAencryption", Required = false)]
        public string RSA
        {
            get { return _rsa; }
            set { _rsa = value; }
        }
        private string _dpapi;
        [TaskAttribute("DPAPIencryption", Required = false)]
        public string DPAPI
        {
            get { return _dpapi; }
            set { _dpapi = value; }
        }
        protected override void ExecuteTask()
        {
            if (checkFileExists())
            {
                if (FilePath.Contains("web.config"))
                {
                    encryptWebConfig();
                    rename();
                }
                else
                {
                    encryptAppConfig();
                    rename();
                }
            }
            else
            {
                throw new BuildException("The .config nominated does not seem to exist");
            }
        }

I have left the encryption methods as optional - this way, you dont need to worry about which type you need, however by doing this I need to set up a default encryption provider. A simple method like the one below will take care of that for me, I have established that I want to use RSA as the default encryption method if no other is specified:

private void chooseEncryptionProvider()
        {
            if (RSA == "true")
            {
                strProvider = "RSAProtectedConfigurationProvider";
            }
            else if (RSA == "" & DPAPI == "")
            {
                strProvider = "RSAProtectedConfigurationProvider";
            }
            else if(DPAPI == "true")
            {
                strProvider = "DataProtectionConfigurationProvider";
            }
        }

I also need to make sure that the file I want to apply encryption to exists in the first place. Again, this is very simple to do:

private bool checkFileExists()
        {
            if (File.Exists(FilePath))
            {
                return true;
            }
            else
            {
                return false;
            }
        }

Now I am all set to get encrypting. I want my task to be able to handle both web and app.config files, so I have two methods to handle each. They are pretty much identical in their operation, the only real difference is down to the way .Net handles web.config files. To encrypt a web.config I have done the following:

private void encryptWebConfig()
        {
            chooseEncryptionProvider();
            try
            {
                Configuration _configFile = WebConfigurationManager.OpenWebConfiguration(FilePath);
                if (_configFile != null)
                {
                    try
                    {
                        ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
                        _section.SectionInformation.ProtectSection(strProvider);
                        _section.SectionInformation.ForceSave = true;
                        _configFile.Save();
                    }
                    catch (Exception e)
                    {
                        throw new BuildException("Failed to encrypt the connection strings sectiion",
                            e.InnerException);
                    }
                }
            }
            catch (Exception e)
            {
                throw new BuildException(e.Message.ToString(), e.InnerException);
            }
        }

You can see where I am choosing the encryption provider at the start of the method. One thing I havent done here is implement any logging. For the sake of an audit trail, it would be a very wise thing to log what sort of encryption is being used. This way, you can look back across the builds you have completed for a specific environment and see which encryption provider was used. The following method encrypts an app.config:

private void encryptAppConfig()
        {
            string path = FilePath.Replace(".config", "");
            chooseEncryptionProvider();
            Configuration _configFile = ConfigurationManager.OpenExeConfiguration(FilePath);
            if (_configFile != null)
            {
                try
                {
                    ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
                    _section.SectionInformation.ProtectSection(strProvider);
                    _section.SectionInformation.ForceSave = true;
                    _configFile.Save();
                }
                catch (Exception e)
                {
                    throw new BuildException("Failed to encrypt the connection strings section"
                        , e.InnerException);
                }
            }
        }

As you can see, it is pretty much the same idea for this operation compared to encrypting a web.config. With all this in place, we now have all we need to automatically encrypt a .config file as part of an automated build. This can be added on to the end of a build target in a NAnt script, or better yet added to its own target to be used as a post build event. One thing I found when using this is that after the .config had been encrypted a .config.config file was created leaving the un-encrypted .config in the same directory. Naturally, this defeats the point so I put the following method together to clean up for me:

private void rename()
        {
            if (File.Exists(FilePath + ".config"))
            {
                string oldname = FilePath;
                string newname = FilePath + ".config";
                File.Delete(FilePath);
                File.Copy(newname, oldname);
                File.Delete(newname);
            }
        }

And there we go, all done. The syntax for the build script is insanely simple - if you dont mind using RSA encryption all you need to do is point it at the .config you want to encrypt:

</target>
  <target name="encrypt">
    <config-encrypt configfile="C:\test\web.config" />
  </target>

I have been using this task for the past week or so, and it has been working like a dream :). The complete code for the task is below:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.IO;
using System.Linq;
using System.Text;
using System.Web;
using System.Web.Configuration;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;

namespace Custom.NantTasks.Encryption
{
    [TaskName("config-encrypt")]
    class Encryption : Task
    {
        private string strProvider;
        private string _filePath;
        [TaskAttribute("configfile", Required = true)]
        public string FilePath
        {
            get { return _filePath; }
            set { _filePath = value; }
        }
        private string _rsa;
        [TaskAttribute("RSAencryption", Required = false)]
        public string RSA
        {
            get { return _rsa; }
            set { _rsa = value; }
        }
        private string _dpapi;
        [TaskAttribute("DPAPIencryption", Required = false)]
        public string DPAPI
        {
            get { return _dpapi; }
            set { _dpapi = value; }
        }
        protected override void ExecuteTask()
        {
            if (checkFileExists())
            {
                if (FilePath.Contains("web.config"))
                {
                    encryptWebConfig();
                    rename();
                }
                else
                {
                    encryptAppConfig();
                    rename();
                }
            }
            else
            {
                throw new BuildException("The .config nominated does not seem to exist");
            }
        }
        private void encryptWebConfig()
        {
            chooseEncryptionProvider();
            try
            {
                Configuration _configFile = WebConfigurationManager.OpenWebConfiguration(FilePath);
                if (_configFile != null)
                {
                    try
                    {
                        ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
                        _section.SectionInformation.ProtectSection(strProvider);
                        _section.SectionInformation.ForceSave = true;
                        _configFile.Save();
                    }
                    catch (Exception e)
                    {
                        throw new BuildException("Failed to encrypt the connection strings sectiion",
                            e.InnerException);
                    }
                }
            }
            catch (Exception e)
            {
                throw new BuildException(e.Message.ToString(), e.InnerException);
            }
        }

        private void encryptAppConfig()
        {
            string path = FilePath.Replace(".config", "");
            chooseEncryptionProvider();
            Configuration _configFile = ConfigurationManager.OpenExeConfiguration(FilePath);
            if (_configFile != null)
            {
                try
                {
                    ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
                    _section.SectionInformation.ProtectSection(strProvider);
                    _section.SectionInformation.ForceSave = true;
                    _configFile.Save();
                }
                catch (Exception e)
                {
                    throw new BuildException("Failed to encrypt the connection strings section"
                        , e.InnerException);
                }
            }
        }

        private void chooseEncryptionProvider()
        {
            if (RSA == "true")
            {
                strProvider = "RSAProtectedConfigurationProvider";
            }
            else if (RSA == "" & DPAPI == "")
            {
                strProvider = "RSAProtectedConfigurationProvider";
            }
            else if(DPAPI == "true")
            {
                strProvider = "DataProtectionConfigurationProvider";
            }
        }

        private bool checkFileExists()
        {
            if (File.Exists(FilePath))
            {
                return true;
            }
            else
            {
                return false;
            }
        }

        private void rename()
        {
            if (File.Exists(FilePath + ".config"))
            {
                string oldname = FilePath;
                string newname = FilePath + ".config";
                File.Delete(FilePath);
                File.Copy(newname, oldname);
                File.Delete(newname);
            }
        }
    }
}

Using log4net with C# and MySql

2009-05-19T13:22:00.000-07:00

Logging your application is a basic unwritten requirement. I cannot think of one application I have ever worked with that does not provide any form of logging, be it a logfile or a log in a database.

Recently at work, I came across a situation where someone described a log as a "nice to have". No, it's not nice to have - its essential to have it. As I was working in .Net (c# to be exact, hence the title), I decided to take a look at log4net. This is a logging framework maintained by the Apache project and is a port of the popular log4j to .Net, I am not going to bore people with an explanation of it here, all I will say about it is that it is a very powerful and flexible logging service. You can check out the official site here: http://logging.apache.org/log4net/index.html.

I spent a few weeks working with log4net (as it apparently wasnt essential to have a log, it was a low priority task). Creating a logfile with it is incredibly easiy, as is logging to the console. There are wide range of methods for creating a log, called appenders, however the most useful for me was appending the log to a database. Whilst log4net is popular, powerful and active, there isnt that much in the way of documentation or howto's, which could put some people off using it, this is especially apparent when logging to a database. The examples that already exist on the interweb, while useful, are sometimes out of date using deprecated syntax.

For my logging scenario, I needed to maintain a log in a MySql table, this compounded my explotation of log4net, as it isnt readily obvious how this can be done. My aim is to use log4net for all of my logging needs, so I want to create an assembly that can be referenced etc wherever I need it. Due to the extremely felxible nature of log4net, this is achievable - the majority of the configuration for it lives within the app.config (or web.config if you are that way inclined). It is possible to programatically configure log4net from within your code - but what would be the point? If you do that, then you loose all of the flexibility offered. One way to highlight this feature - right now I need to log to a MySql table, but what about in future if we decide to migrate to another DB? If I configured my logger programtically, I would then need to rewrite code. Whereas if I configured it via a .config file, all I need to do is deploy that file wherever it needs to go...

Anyway, enough waffle - you are all grownups and have myriad needs. Lets take a look at how I got log4net logging to my MySql table.

Setting up log4net to log to the console or to a file is dead simple, there is very little taking place in the code, most of it is done in the config file. The same can go for logging to a db, but with one small difference - if you rely on the config file to control your logging when using a database you have to be comfortable with the fact that your connection string will be visible. I am not sure how much of a security risk that is for everyone else, but for me it wasnt something I was willing to do at all. Even if you use a seperate database for logging, you are still exposing a lot of information about your infrastructure here. So, instead of placing the connection string in the log4net config, I am going to place it in the connectionStrings section and encrypt it. Once this is done, I need to set the connection programatically. This isnt as easy as you would first imagine (well, at least I didnt find it that easy), whilst there is a lot of documentation on what log4net can do, I found it quite tricky to figure out how to make it work with MySql - and even when I had I still had one small issue... Anyway, on to getting this setup.

All of my config for my logger is going to live in my app.config - inlcuded my encrypted db connection string. I need to get this string and then provide it to log4net, this bit is done in the code. It sounds simple but there are a number of steps I need to take first. To start with I need to get the default repository that my logger is going to use with log4net:

private Hierarchy hierachy()



        {



            Hierarchy h = (Hierarchy)log4net.LogManager.GetRepository();



            return h;



        }

It is possible to use different configuration repositories with log4net - but I dont need to so I havent had to worry about this. To be honest, I am not sure that a lot of people will need to either. Once this has been done, we need to create an ADOAppeneder - it should also be pointed out by now that I am using the MySql .Net connector to work with my db.

private AdoNetAppender adoAppender()



        {



            Hierarchy myHierarchy = hierachy();



            if (myHierarchy != null)



            {



                AdoNetAppender adoApp = (AdoNetAppender)myHierarchy.Root.GetAppender("ADONetAppender");



                return adoApp;



            }



            return null;



        }

With this code, I am able to get the section of my .config file that holds all of the config for logging to a db through log4net, log4net relies on one or more 'appenders' being configured for use - my db appender is simply called 'ADONetAppender', so I have specified this in the above method. With this done, I am now ready to configure my appender programtically:

private void createAdoSettings(string dbConn)



        {



            AdoNetAppender myAppender = adoAppender();



            if (myAppender != null)



            {



                myAppender.ConnectionString = dbConn;



                myAppender.UseTransactions = true;



                myAppender.ActivateOptions();



            }



        }

After all of this, we are done. All I need to do is tell my logging class what my connection string is. This isnt so difficult (thankfully, is very easy):

string connstring = ConfigurationManager.ConnectionStrings["SQLLogDB"].ConnectionString.ToString();



            SetUpLog _setup = new SetUpLog(connstring);

Cant get any simpler really :). All I need to do is drop this into any class I want to log from and I am set. There is only one other addition I need to make, and that is to implement the ILog interface. This is the most important bit, regardless of any db logging, without it you cant log anything at all! I got a nice little gem from here: http://www.ondotnet.com/pub/a/dotnet/2003/06/16/log4net.html that give a good tip on implementing the interface. Basically, popping this line in at the start of your class ensures that log4net reports the name of the class duing the logging process:

protected static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

This is so useful and provides a very concise log. With all this in place, we are now able to use log4net to log our application. The following code is the test app I wrote when to prove that it works:

using Logger.Setup;



using System;



using System.Configuration;



using System.Collections.Generic;



using System.Linq;



using System.Text;







namespace TestClient



{



    class Program



    {



        protected static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);



 



        static void Main(string[] args)



        {



            Program test = new Program();



            test.logtest();   



        }



        private void logtest()



        {



            string connstring = ConfigurationManager.ConnectionStrings["LogDB"].ConnectionString.ToString();



            SetUpLog _setup = new SetUpLog(connstring);     



            log.Info("Hello there");



        }



    }



}

On execution, it provides the following on the command line:

2009-05-23 23:21:37,536 [12260] INFO  Hello there 23 TestClient.Program

You can also see how I am specifying the connection string, which is also held in the .config file. This is driven by my .config file, it is just a console appender and lives inside the log4net section:

<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">



      <layout type="log4net.Layout.PatternLayout">



        <conversionPattern value="%date [%thread] %-5level %message%newline %L %C" />



      </layout>



    </appender>

Now I know it works, I can get down to logging the same information to my db, first I create my schema:

CREATE TABLE logdb.mylog(



  `DateTime` DATETIME DEFAULT NULL,



  Thread VARCHAR (20) DEFAULT NULL,



  Level VARCHAR (20) DEFAULT NULL,



  Message VARCHAR (20) DEFAULT NULL,



  Exception VARCHAR (20) DEFAULT NULL,



  line VARCHAR (20) DEFAULT NULL,



  object_ref VARCHAR (20) DEFAULT NULL,



  taskID VARCHAR (20) DEFAULT NULL



)



ENGINE = INNODB



CHARACTER SET latin1



COLLATE latin1_swedish_ci;

The object_ref and taskID columns are things I want to capture as part of my wider logging process - they dont really have anything to do with log4net or its configuration. With my schema in place all I need to do now is configure log4net to log my messages to my new table. This can be a bit frustrating when working with MySql - if you get something wrong in the config you will get an error message, but they dont always seem to be that helpful to me. For instance, log4net requires you to define a connection type in order to log to a db. When I first started working with log4net, my connection type was incorrect which resulted in an error. However, the error I got complained about connection strings being null - not that there was an error with the connection type I had configured. Granted, they are closely related, but it would have just been nice if I get an error back telling me that my connection type was wrong... Anyway, I got passed it. When configuring log4net there are a couple sections that you need to work with. I am going to assume that if you are reading this then you have already looked at the log4net site and looked at how it is configured. I am not going to explain how to configure log4net here in general, if you want to see how this works then there are a lot of other articles out there that do this well, one good place to start on the topic of configuration is the log4net site - even though it is very general it will give you an idea of how all the appenders look: http://logging.apache.org/log4net/release/manual/configuration.html

Now that my schema is in place and my logic is complete, all I need to do now is configure a log4net appender. Appenders in log4net control how information is formatted as it is logged, I am going to use an ADO appender and it looks like this:

<appender name="ADONetAppender" type="log4net.Appender.ADONetAppender">
      <connectionType value="MySql.Data.MySqlClient.MySqlConnection, MySql.Data, Version=6.0.3.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" />
      <commandText value="insert into mylog (thread,line,object_ref,taskID,message,level,datetime)
                   values(?thread,?line,?objectref,?taskid,?message,?level,?log_date)" />
      <parameter>
        <parameterName value="?thread"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%thread"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?line"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%L"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?objectref"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%C"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?taskid"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%property{taskid}"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?message"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%message"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?level"/>
        <dbType value="String"/>
        <size value="20"/>
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%level"/>
        </layout>
      </parameter>
      <parameter>
        <parameterName value="?log_date" />
        <dbType value="DateTime" />
        <layout type="log4net.Layout.RawTimeStampLayout"/>
      </parameter>
    </appender>

The most notable thing about my config here is the use of a property. This lets me capture custom information for my log and place this in my db. This is really useful, I am sure you can all understand why and goes to show the power of log4net. With all of this in place, I am now able to log to my db - but with one small caveat, I think I have found a problem with log4net when using an ADO appender with MySql. For some reason, log4net generates an error telling me that there is a null system exception, but carries on and logs the data to the db anyway. I spent a long time trying to figure out what the problem was, but couldnt spot it. So, after some advice, I started working with the log4net source to spot where the error was being generated. Stepping through the code, I found that log4net (when used with the MySql .Net connector) seems to loose the connection string, but logs data anyway. I couldnt replicate this with SQL Server and have still been unable to resolve the error. I dont like errors hanging around like this, the data is logged, but I get this nasty message.... I have posted my findings to the log4net support group, but have not had anything back about it really - which is a shame. However, in saying that, log4net is still a truly useful tool to use in any project that requires any form of logging.

NAnt, .Net, Sandcastle and Documentation

2009-05-09T05:03:00.000-07:00

Documentation is great, it makes people feel good about themselves. After they have created a particularly cunning peice of work, it's nice to throw together some documentation so that others can see your genius. But wait, that means you need to go through all your work and come up with some documentation of your own! Gasp, shock - horror!!

Lets face it, for a lot of people, creating standardised documentation after they have finished their project etc isnt at the fore front of their minds. This is a shame, because with flippancy aside, people do really like documentation. It enables developers to get on with their work without asking people for info on an API etc and it lets managers see that there is "stuff being done with documentation". And dont forget, standardisation is key here - you dont want a hundred different developers all creating documentation in different ways, it would be a nightmare.

There are a lot of ways documentation can be generated at build time. One popular way is to use NDoc and this is catered for within NAnt. However, this doesnt support .Net 2.0 let alone 3.5 and it hasnt seen a release since 2005... So, there appears to be a massive gap for generation of documentation, that is if you don't count Sandcastle. This is a fairly useful, if slightly complex method of generating documentation. There are lots of examples on how to use Sandcastle via batch files and powershell scripts (the version I downloaded came packed with them). In an effort to understand the generation process, I decided to put together a NAnt task so this could be automated as part of continuous integration.

Sandcastle itself is not really one application. Right now, Sandcastle is a number of programs working together. It consits of:

MRefBuilder.exe
XslTransform.exe
BuildAssembler.exe
CHMBuilder.exe
DBCSFix.exe

Each of these programs are used in turn to generate first XML documentation followed by CHM (should you need it.). The following examples give an indication of how you could put a NAnt task together to make use of these programs for all your documentation needs.

All we need to do is create a suite of tasks that represent each of these programs. Essentially, all we are doing here is capthuring the arguments we need to run the apps themselves, there isnt much going on in task per se. For example, here is the complete code for MRefBuilder:

using System;
using System.Collections.Generic;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;
using Custom.NantTasks.Common;

namespace Custom.NantTasks.Sandcastle
{
    /// <summary>
    /// NAnt task to make use of Sandcastle from MS
    /// </summary>
    [TaskName("mrefbuilder")]
    class MRefBuilder : ExternalProgramBase
    {
        private string _assemblyPath;
        [TaskAttribute("assembly-path", Required = true)]
        public string AssemblyPath
        {
            get { return _assemblyPath; }
            set { _assemblyPath = value; }
        }
        private string _outputPath;
        [TaskAttribute("output-path", Required = true)]
        public string OutPutPath
        {
            get { return _outputPath; }
            set { _outputPath = value; }
        }
        private string _executable;
        [TaskAttribute("executable", Required = true)]
        public string Executable
        {
            get { return _executable; }
            set { _executable = value; }
        }
        public override string ExeName
        {
            get
            {
                return _executable;
            }
            set
            {
                base.ExeName = value;
            }
        }
        public override string ProgramArguments
        {
            get { return AssemblyPath + " /out:" + OutPutPath; }
        }
        protected override void ExecuteTask()
        {
            Log(Level.Info, "Running MRefBuilder");
            Task();
        }

        private void Task()
        {
            try
            {
                base.ExecuteTask();
            }
            catch (Exception e)
            {
                throw new BuildException(e.Message);
            }
        }
    }
}

As you can see, this is pretty much just an arguments gathering exercise. Things become a little trickier for XslTransform. This app specifies multiple transformations, so we need to be able to specify at least one transformation in the task itself - but reserve the ability to specify a second. I am not going to go into exactly how Sandcastle works here, all I would be doing is copying information. For an in depth explanation of how Sandcastle works, take a look at the Sandcastle blog. So, for the XslTransform task, I simply specify two attributes for the .xsl files:

private string _xslPath;
        [TaskAttribute("xsl-path", Required = true)]
        public string XslPath
        {
            get { return _xslPath; }
            set { _xslPath = value; }
        }
        private string _xslPath2;
        [TaskAttribute("xsl-path2", Required = false)]
        public string XslPath2
        {
            get { return _xslPath2; }
            set { _xslPath2 = value; }
        }

You may wish to use more than one .xsl - I don't, the example batch file in the version I downloaded didnt seem to make use of more than two. However, if you need to add more then its a very simple task to add this to the task itself. Once the arguments have been captured, we need to then use them with the app itself. We need to be able to provide at least one .xsl in the argument we use with the app and if a second is specified, then we need to be able to provide this also. To achieve this, I used the following:

private string checkXSLPath()
        {
            string _arg;
            if (XslPath2 == null)
            {
                _arg = "/xsl:\"" + XslPath + "\" " + InputFile + " /out:" + OutputPath;
                Log(Level.Info, _arg);
                return _arg;
            }
            else
            {
                _arg = "/xsl:\"" + XslPath + "\" " + "/xsl:\""
                       + XslPath2 + "\" " + InputFile + " /out:" + OutputPath;
                Log(Level.Info, _arg);
                return _arg;
            }
       
        }

All this does is simply check to see if a second xsl file has been specified. If it has, then it returns an argument that makes use of it, if not, then it returns an argument that doesnt. The full source looks like this:

using System;
using System.Collections.Generic;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;

namespace Custom.NantTasks.Sandcastle
{
    [TaskName("xsltransform")]
    class XslTransform : ExternalProgramBase
    {
        private string _xslPath;
        [TaskAttribute("xsl-path", Required = true)]
        public string XslPath
        {
            get { return _xslPath; }
            set { _xslPath = value; }
        }
        private string _xslPath2;
        [TaskAttribute("xsl-path2", Required = false)]
        public string XslPath2
        {
            get { return _xslPath2; }
            set { _xslPath2 = value; }
        }
        private string _inputFile;
        [TaskAttribute("inputfile", Required = true)]
        public string InputFile
        {
            get { return _inputFile; }
            set { _inputFile = value; }
        }
        private string _outPutPath;
        [TaskAttribute("output", Required = true)]
        public string OutputPath
        {
            get { return _outPutPath; }
            set { _outPutPath = value; }
        }
        private string _executable;
        [TaskAttribute("executable", Required = true)]
        public string Executable
        {
            get { return _executable; }
            set { _executable = value; }
        }
        public override string ExeName
        {
            get
            {
                return _executable;
            }
            set
            {
                base.ExeName = value;
            }
        }
        public override string ProgramArguments
        {
            get { return checkXSLPath(); }
        }
        protected override void ExecuteTask()
        {
            Log(Level.Info, "Starting XslTransform");
            Task();
        }

        private void Task()
        {
            try
            {
                base.ExecuteTask();
            }
            catch (Exception e)
            {
                throw new BuildException(e.Message);
            }
        }
        private string checkXSLPath()
        {
            string _arg;
            if (XslPath2 == null)
            {
                _arg = "/xsl:\"" + XslPath + "\" " + InputFile + " /out:" + OutputPath;
                Log(Level.Info, _arg);
                return _arg;
            }
            else
            {
                _arg = "/xsl:\"" + XslPath + "\" " + "/xsl:\""
                       + XslPath2 + "\" " + InputFile + " /out:" + OutputPath;
                Log(Level.Info, _arg);
                return _arg;
            }
        
        }
    }
}

Looking good so far. Now we have two NAnt tasks that will generate documentation as xml and then transform it. Now we get to the meat of the whole process, BuildAssembler. Again, I am not going to go into detail about what this does here - instead, check the Sandcastle blog entry on BuildAssembler here for a complete explanation. Whilst the process behind BuildAssembler is very involved, the NAnt task for it is quite simple, the complete source is below:

using System;
using System.Collections.Generic;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;

namespace Custom.NantTasks.Sandcastle
{
    [TaskName("buildassembler")]
    class BuildAssembler : ExternalProgramBase
    {
        private string _manifest;
        [TaskAttribute("manifest", Required = true)]
        public string Manifest
        {
            get { return _manifest; }
            set { _manifest = value; }
        }
        private string _config;
        [TaskAttribute("config", Required = true)]
        public string Config
        {
            get { return _config; }
            set { _config = value; }
        }
        private string _executable;
        [TaskAttribute("executable", Required = true)]
        public string Executable
        {
            get { return _executable; }
            set { _executable = value; }
        }
        public override string ExeName
        {
            get
            {
                return _executable;
            }
            set
            {
                base.ExeName = value;
            }
        }
        public override string ProgramArguments
        {
            get { return " /config:\"" + Config + "\" " + "\"" + Manifest + "\""; }
        }
        protected override void ExecuteTask()
        {
            Log(Level.Info, "Starting BuildAssember");
            base.ExecuteTask();
        }
    }
}

Essentially, all we are doing is giving BuildAssembler a config file to work with as well as a manifest file we generated with the previous task, XslTransform. Now even though there are more steps to go through, we need to start looking at the NAnt script itself. On its own, Sandcastle wont do everything you want, there are things you need to initiate, some of these are the creation of directories as well as copying files etc. Lets take a look at the NAnt script, at this stage it should look something similar to this:

<target name="Sandcastle">
    <echo message="Trying out my Sandcastle task" />
    <echo message="Checking for output directory.." />
    <mrefbuilder assembly-path="Custom.NantTasks.dll"
                 output-path="C:\somefolder\sandcastletest.xml"
                 executable="C:\Program Files\Sandcastle\ProductionTools\MrefBuilder.exe" />
    <xsltransform xsl-path="C:\Program Files\Sandcastle\ProductionTransforms\ApplyVsDocModel.xsl"
                  xsl-path2="C:\Program Files\Sandcastle\ProductionTransforms\AddFriendlyFilenames.xsl"
                  inputfile="C:\somefolder\sandcastletest.xml"
                  output="C:\somefolder\refelction.xml"
                  executable="C:\Program Files\Sandcastle\ProductionTools\xsltransform.exe" />
    <xsltransform xsl-path="C:\Program Files\Sandcastle\ProductionTransforms\ReflectionToManifest.xsl"
                  inputfile="C:\somefolder\refelction.xml"
                  output="C:\somefolder\manifest.xml"
                  executable="C:\Program Files\Sandcastle\ProductionTools\xsltransform.exe" />
    <mkdir dir="C:\somefolder\html" />
    <mkdir dir="C:\somefolder\icons" />
    <copy todir="C:\somefolder\icons">
      <fileset basedir="C:\Program Files\Sandcastle\Presentation\vs2005\icons">
        <include name="*" />
      </fileset>
    </copy>

    <mkdir dir="C:\somefolder\scripts" />
    <copy todir="C:\somefolder\scripts">
      <fileset basedir="C:\Program Files\Sandcastle\Presentation\vs2005\scripts">
        <include name="*" />
      </fileset>
    </copy>
    <mkdir dir="C:\somefolder\styles" />
    <copy todir="C:\somefolder\styles">
      <fileset basedir="C:\Program Files\Sandcastle\Presentation\vs2005\styles">
        <include name="*" />
      </fileset>
    </copy>
    <mkdir dir="C:\somefolder\media" />
    <mkdir dir="C:\somefolder\intellisense" />
    <buildassembler config="C:\Program Files\Sandcastle\Presentation\vs2005\configuration\sandcastle.config"
                    manifest="C:\somefolder\manifest.xml"
                    executable="C:\Program Files\Sandcastle\ProductionTools\buildassembler.exe" />
  </target>

As you can see, my script is very static. It specifies one model for Sandcastle (VS2005) and specifies one assembly to generate from. Also, as you can see all of the paths are static - but this doesnt stop you from creating a generic task to handle multiple assemblies as part of a CI process. Points to note in this script - simply the creation and population of directories prior to using BuildAssembler. These come directly from the location Sandcastle has been installed to (or from wherever you will choose to store them).

So, we are nearly there. By now, we have xml documentation generated, following the provided scripts in the Sandcastle download, the next step is to create some CHM file. Personally, I dont really need .chm files generated - I want my help files to go straight into a website/wiki.... To do that though, requires a bit more thought and code than I cover here. So stay tuned!!

Cycle paths, great...

2009-05-03T11:39:00.000-07:00

I have been getting on my bike lately. This isnt such a novel thing for me, it was a past time for me prior to my move up to Scotland. Cycling up here is a lot different to my previous venue, the weather, for one thing, is completely different. Down south I could be content in the knowledge that it would be relatively sunny from March to September and also had the added benefit of being on the doorstep to a national park. In Edinburgh though, things are different by quite some degree, for starters, I have to travel quite a bit before I get to anywhere interesting. And as I dont have a car, let alone a license, it means a period of cycling through a rather boring Edinburgh.

But, thats all forgotten once you hit one of the cycle routes that cross through Edinburgh. I was out near a place called Auchendinny a few days ago and managed to get myself lost (wasnt on a cycle route), on the way back into Edinburgh I had the misfortune to pass through Bonnyrigg. I have never been there before and if ever I go back I will be sure to spend as little time as possible there. I am sure there are a lot of lovely people there, but unfortunately the only natives I met decided to through a vodka bottle at me and shredded my front tire in the process.

Charming.

It was in such a poor state, I had to get a lift back into Edinburgh. Due to Easter, it meant that the buses were on a crap schedule - by pure chance I was lucky enough to find a cab, not only an available one but one who would admit my rather mucky bike and me in it. I didnt catch his name, but he was a life saver and has my undying thanks.

And no, that wasnt in Bonnyrigg, it was a few miles down the road in Dalkeith!

Delphi.Net and the Internet

2008-11-12T04:24:00.000-08:00

A few days ago, I posted about Delphi 2009. I was pleased with the additions that Codegear had made to the suite and speculated that it could be a possible migration route from pure Delphi over to .Net. I have had a think about this now and have sort of changed my mind. Using Delphi 2009 it seems to me that mature code written in Delphi could be used in any other .Net application - this includes Net powered websites.

So, instead of being a migration to .Net from Delphi, it is more along the lines of making use of good, solid Delphi code that has been developed over a number of years in a .Net envronment. I really like this idea and am interested to see how many Delphi developers would look into this. It certainly opens up a whole new world of possibility.

Delphi and .Net

2008-11-07T06:32:00.000-08:00

I have been looking at Delphi 2009 lately. Pascal was the first programming language I ever learned, I would imagine that this is the same for many people my age, so to take a look at Delphi after so long is a bit nostalgic for me.

However, in saying that, I have always stayed away from Delphi as a development tool - a lot of the work I do needs to be cross platform and this is something that hasnt always been do-able with Delphi. However, version 2009 not only includes updates to Delphi.Net, but also includes Generics and anyonmous methods - it also adds user definable build configurations. This might seem a petty thing to be pleased about, but for anyone developing or working with cross platform soloutions, this is a godsend.

At first glance, the .Net abilities of Delphi 2009 seem to suggest that Codegear would like people to migrate from regular Delphi over to .Net - the IDE supports developing VB.Net and C# applications as well as Delphi and Delphi.Net apps. Personally, I am a little surprised that Microsoft hasn't tried its own version of Delphi.Net and called it D# or something - there are so many systems out in the wild using Delphi in a Windows environment.

As it stands right now, there is nothing stopping you from migrating core logic into C#, for instance, and using this in a Delphi.Net UI. I think this is a nice stepping stone for systems written in Delphi to make the hop from the desktop and onto the web, but we shall just have to wait and see. There is a wide dirth of new languages soon to be available under the .Net framework like Python, Ruby and F#, so it looks like the trend is moving towards dynamic languages for the time being.

Generating SQL Server Documentation with NAnt and RedGate

2008-10-04T10:08:00.000-07:00

The other day I posted a method by which you can automatically sync schema from one MS SQL Server db to another. While I was looking at the RedGate tools, I found a handy utility called SQL Doc. This doesnt have an api (and doesnt really need one) it consits of a GUI and a commandline client, which is good.

Basically, this app allows you to create a set of documentation on any given db including the setup scripts needed to replicate that particular db. It can output this report as a set of web pages, a .chm or a .doc. I thought that this was pretty cool, it represents a really good way of documenting the db at certain points through out its life. One good use of this tool would be to generate documentation prior to a new release (or just after). That way, you can maintain a handy record of the changes going into your db that can be published on your intranet/wiki etc.

Of course, as a GUI tool, this isnt so great for CI, so whacking the commandline parameters into a NAnt script will make it slot into your automated process nicely. Plus, it's an easy win - the task itself is very easy to write:

using System;
using System.Text;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;

namespace Custom.NantTasks.RedGate
{
    [TaskName("sqldoc")]
    class SQLDoc : ExternalProgramBase
    {
        private string _projectPath;
        [TaskAttribute("project", Required = true)]
        public string ProjectPath
        {
            get { return _projectPath; }
            set { _projectPath = value; }
        }
        private string _reportType;
        [TaskAttribute("report-type", Required = true)]
        public string ReportType
        {
            get { return _reportType; }
            set { _reportType = value; }
        }
        private string _password;
        [TaskAttribute("password", Required = true)]
        public string Password
        {
            get { return _password; }
            set { _password = value; }
        }
        private string _exePath;
        [TaskAttribute("app-path", Required = true)]
        public string ExePath
        {
            get { return _exePath; }
            set { _exePath = value; }
        }
        public override string ExeName
        {
            get
            {
                return _exePath;
            }
            set
            {
                ExeName = value;
            }
        }
      
        public override string ProgramArguments
        {
            get { return "/project:\"" + ProjectPath + "\"" + " /password:" + Password +" /filetype:" + ReportType; }
        }
        protected override void ExecuteTask()
        {
            Log(Level.Info, "Generating SQL documentation");
            Log(Level.Info, "Running the following command: " + ProgramArguments.ToString());
            base.ExecuteTask();
        }
    }
}

And thats it!! There are, however, a few caveats. In order to automate this, you first need to set up a project file through the GUI. Unfortunately, there doesnt seem to be a commandline route to do this. Once you have configured the project and saved it, you basically get an XML file - which is very handy as you could use NAnt to alter the values held within it (I havent done this). The build file would look a little something like this:

<target name="doc">
  <sqldoc project="C:\WidGetDev.sqldoc" password="password" report-type="html" app-path="C:\Program Files\Red Gate\SQL Doc 2\SQLDoc.exe" />
  </target>

Unfortunately, there is no way to specify a username on the commandline, only a password, which I guess is good enough. Once done, you can run this as a NAnt task pretty much anywhere. I did this on Vista, so I got some interesting security notifications pop up. I also got a very annoying bug following a sucessful build. For some reason, SQL Doc would fail to exit correctly (exiting on code 77, which means that the user does not have permission), however this just went away for me in the end... which was odd. Regardless, once the task has ran and completed, you will be able to pick up your documentation, SQL Doc places it in a location specified during the project creation, but you can change this whenever you like. After the task is run, just do what ever you fancy with the results, my idea was to post them to a wiki, but you should be able to attach them as a build artefact in which ever CI server you are using (if any).

Sync Schema with NAnt and RedGate SQL Compare

2008-10-03T08:10:00.000-07:00

This was a problem that was bothering me a little throughout last year. On my previous blog I had a very basic NAnt task that made use of RedGate's SQL Compare API. RedGate makes some fantastic tools for use with MS SQL Server, if you have to spend time working on multiple databases, then I can't recommend it enough. For more infor mation, pop over to their website: http://www.redgate.com/.

First up, you need to have a little think about the problem at hand - you have a number of databases that you need to sync. SQL Compare gives you a nice GUI to do all of this, which is fine if you only have one db to worry about (or at a stretch, a couple.) but if you have a large amount of db's spread across multiple environments this will become a problem very quickly. To get around this problem, you can make use of the command line version of SQL Compare or you can do what I did and make use of the API to automate things just a little further. One clear benefit I have noticed so far is that using the API over the command line version results in a quicker operation.

Another thing to consider is this: If you have more than one db to manage, you are going to have to put the login credentials somewhere. This will potentially panic your IT manager or administrator - they dont really want login credentials floating around on machines, which is a very valid and fair point. Froms the example I have below, you will see that I am managing my login credentials by placing them in NAnt's app.config. This will send a shiver of fear down some peoples spines, as this means that they are there for all to see in plain text - but, dont panic. It is a very simple thing to encrypt them so that no one can see them. Just plonk all your db credentials in there, encrypt it and make sure that this .config is deployed correctly when ever the details change.

I am sure there will be other considerations to take into account depending on your own setup (there always is). But for me, this is all I needed. So, lets move on - what do I need to capture in my NAnt build script? Well, if all the db credentials are stored away in a .config file then all I need to do is retrieve them by name. This means that in my NAnt script, I just need to specify the source and destination db's based on the key value, e.g;

<add value="db-server-name, db-name, username, password" key="test">

It's up to you if you want this to be in plain text in NAnt's config file, personally I would encrypt it - but thats just me.

Now, onto the task. As I said above, I am only getting two attributes: the source and destination database names. This means the task would look a little like this:

[TaskName("schema-sync")]
    public class SchemaSync : Task
    {
        public Sync sync = new Sync();
        private string _sourceDBName;
        [TaskAttribute("source", Required = true)]
        public string SourceDBName
        {
            get { return _sourceDBName; }
            set { _sourceDBName = value; }
        }
        private string _destinationDBName;
        [TaskAttribute("destination", Required = true)]
        public string DestinationDBName
        {
            get { return _destinationDBName; }
            set { _destinationDBName = value; }
        }

Now, we get onto the good stuff - getting the sync underway!

I am going to present this in the way I figured it out - I am fairly sure that a lot of people reading this will want to rejig it to fit there own needs. Lets start off getting a collection of differences between two databases:

private Differences GetDifferences(string dbServerNameSource,
                                               string dbUserNameSource,
                                               string dbPasswordSource,
                                               string dbNameSource,
                                               string dbServerNameDest,
                                               string dbUsernameDest,
                                               string dbPasswordDest,
                                               string dbNameDest)
        {
            Differences differences = null;
            Database db1 = new Database();
            Database db2 = new Database();
            db1.Register(new ConnectionProperties(dbServerNameSource,
                                                  dbNameSource,
                                                  dbUserNameSource,
                                                  dbPasswordSource),
                                                  Options.Default);
            db2.Register(new ConnectionProperties(dbServerNameDest,
                                                  dbNameDest,
                                                  dbUsernameDest,
                                                  dbPasswordDest),
                                                  Options.Default);
            differences = db1.CompareWith(db2, Options.Default);
            foreach (Difference difference in differences)
            {
                difference.Selected = ExcludeObject(difference);
            }
            return differences;
        }

This method returns a collection of differences between two database's as supplied through the NAnt script. Note that this doesnt actually affect any changes at all on your db's, all it does it find out just how many differences there are. Next, we want to build the SQL needed to sync the schema from DB1 to DB2, this is fairly simple:

private ExecutionBlock ScriptDifferences(Differences differences)
        {
            Work work = new Work();
            work.BuildFromDifferences(differences, Options.Default, true);
            ExecutionBlock block = work.ExecutionBlock;
            return block;
        }

The returned block now contains the SQL script that will alter the destination db. To execute this SQL, we need the following:

private BlockExecutor SyncDifferences(string dbServerNameDest,
                                      string dbNameDest,
                                      string dbUserNameDest,
                                      string dbPasswordDest,
                                      ExecutionBlock block)
        {
            BlockExecutor executor = new BlockExecutor();
            executor.ExecuteBlock(block,
                                  dbServerNameDest,
                                  dbNameDest,
                                  true,
                                  dbUserNameDest,
                                  dbPasswordDest);
            return executor;
        }

And thats it, this will effectively sync db1 to db2 for you. Incidentally, this seemed to me to be a lot simpler that fudging a script together using the commandline Compare app... You will notice that I am also filtering out some database's from the compare it self, the following will do this for you:


        private static bool ExcludeObject(Difference difference)
        {
            if (difference.DatabaseObjectType != ObjectType.User &&
                difference.DatabaseObjectType != ObjectType.Role &&
                difference.DatabaseObjectType != ObjectType.Queue &&
                difference.DatabaseObjectType != ObjectType.Service &&
                SpecialObjects(difference) == false)
            {
                return true;
            }
            return false;
        }
        private static bool SpecialObjects(Difference difference)
        {
            if ((difference.DatabaseObjectType == ObjectType.Table &&
                 difference.Name.ToLower().StartsWith("[dbo].[aspnet_sql"))
                (difference.DatabaseObjectType == ObjectType.StoredProcedure &&
                 difference.Name.ToLower().StartsWith("[dbo].[aspnet_sql"))
                (difference.DatabaseObjectType == ObjectType.StoredProcedure &&
                 difference.Name.ToLower().StartsWith("[dbo].[sqlquery")))
            {
                return true;
            }
            return false;
        }

To tie it all up, I have one public method, which is called from my task class:

public object SyncDB(string dbServerNameSource,
                           string dbUserNameSource,
                           string dbPasswordSource,
                           string dbNameSource,
                           string dbServerNameDest,
                           string dbUserNameDest,
                           string dbPasswordDest,
                           string dbNameDest)
        {
            object obj = null;
            Differences diff = GetDifferences(dbServerNameSource,
                                         dbUserNameSource,
                                         dbPasswordSource,
                                         dbNameSource,
                                         dbServerNameDest,
                                         dbUserNameDest,
                                         dbPasswordDest,
                                         dbNameDest);
            ExecutionBlock script = ScriptDifferences(diff);

            obj = SyncDifferences(dbServerNameDest,
                            dbNameDest,
                            dbUserNameDest,
                            dbPasswordDest,
                            script);
            return obj;
        }

Et voila, we now have enough code to automatically sync schema from one db to another. All that is left to do is to fit this all into my NAnt task. Below is the complete code I used:

using NAnt.Core;
using NAnt.Core.Attributes;
using NAnt.Core.Tasks;
using NAnt.Core.Util;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Text;
using RedGate.Shared.SQL.ExecutionBlock;
using RedGate.SQLCompare.Engine;
using Custom.NantTasks.RedGate.Compare;

namespace Custom.NantTasks
{
    [TaskName("schema-sync")]
    public class SchemaSync : Task
    {
        public Sync sync = new Sync();
        private string _sourceDBName;
        [TaskAttribute("source", Required = true)]
        public string SourceDBName
        {
            get { return _sourceDBName; }
            set { _sourceDBName = value; }
        }
        private string _destinationDBName;
        [TaskAttribute("destination", Required = true)]
        public string DestinationDBName
        {
            get { return _destinationDBName; }
            set { _destinationDBName = value; }
        }
        protected override void ExecuteTask()
        {
            string dbServerNameSource;
            string dbNameSource;
            string dbUserNameSource;
            string dbPasswordSource;
            GetSourceDetails(out dbServerNameSource,
                             out dbNameSource,
                             out dbUserNameSource,
                             out dbPasswordSource);
            string dbServerNameDest;
            string dbNameDest;
            string dbUserNameDest;
            string dbPasswordDest;
            GetDestDetails(out dbServerNameDest,
                           out dbNameDest,
                           out dbUserNameDest,
                           out dbPasswordDest);
            try
            {
                SyncDB(dbServerNameSource,
                       dbNameSource,
                       dbUserNameSource,
                       dbPasswordSource,
                       dbServerNameDest,
                       dbNameDest,
                       dbUserNameDest,
                       dbPasswordDest);
            }
            catch (Exception e)
            {
                Log(Level.Error, "Sync failed");
                throw new BuildException(e.Message);
            }
        }

        private void SyncDB(string dbServerNameSource,
                            string dbNameSource,
                            string dbUserNameSource,
                            string dbPasswordSource,
                            string dbServerNameDest,
                            string dbNameDest,
                            string dbUserNameDest,
            string dbPasswordDest)
        {
   
            Log(Level.Info, "Syncing {0} to {1}", SourceDBName, DestinationDBName);
            try
            {
                sync.SyncDB(dbServerNameSource,
                            dbUserNameSource,
                            dbPasswordSource,
                            dbNameSource,
                            dbServerNameDest,
                            dbUserNameDest,
                            dbPasswordDest,
                            dbNameDest);
                Log(Level.Info, "Sync complete");
            }
            catch (Exception e)
            {
                throw new BuildException(e.Message);
            }
        }

        private void GetDestDetails(out string dbServerNameDest,
                                    out string dbNameDest,
                                    out string dbUserNameDest,
                                    out string dbPasswordDest)
        {
            Log(Level.Info, "Getting credentials for: " + DestinationDBName);
            string[] destDetails =
                ConfigurationManager.AppSettings.GetValues(DestinationDBName)[0].ToString().Split(',');
            dbServerNameDest = destDetails[0].Trim();
            dbNameDest = destDetails[1].Trim();
            dbUserNameDest = destDetails[2].Trim();
            dbPasswordDest = destDetails[3].Trim();
        }

        private void GetSourceDetails(out string dbServerNameSource,
                                      out string dbNameSource,
                                      out string dbUserNameSource,
                                      out string dbPasswordSource)
        {
            string[] sourceDetails =
                ConfigurationManager.AppSettings.GetValues(SourceDBName)[0].ToString().Split(',');
            Log(Level.Info, "Getting credentials for: " + SourceDBName);
            dbServerNameSource = sourceDetails[0].Trim();
            dbNameSource = sourceDetails[1].Trim();
            dbUserNameSource = sourceDetails[2].Trim();
            dbPasswordSource = sourceDetails[3].Trim();
        }
    }
}

GetDestinationDetails() and GetSourceDetails() handle the retrieval of db credentials from NAnt's .config file. To put this new task into use, the build script would look a little like this:

<?xml version="1.0" encoding="utf-8" ?>
<project name="Test" default="test">
  <target name="Test">
      <schema-sync source="WidGetDev" destination="WidGetLive" />
  </target>
</project>

If an error is encountered, then this will be thrown as a build exception and fail the build (which is nice). If you are syncing a number of db's at one time it means the operation will stop at the last successful sync. If you have a number of db's to sync, then it would be a good idea to us NAnt's foreach over a list of db's that you want to sync. This lets you skip over anything you dont want to go from the source to the destination (like a db full of environment variables for instance).

And thats pretty much it, one simple and fairly quick way to use the power of RedGate's SQL Compare engine with NAnt. Thus giving you a nice opportunity to use CI servers like CruiseControl.Net or TeamCity etc.

Below is the complete code for the schema compare:

using NAnt.Core;
using NAnt.Core.Attributes;
using NAnt.Core.Tasks;
using NAnt.Core.Util;
using RedGate.Shared.SQL.ExecutionBlock;
using RedGate.SQLCompare.Engine;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Custom.NantTasks.RedGate.Compare
{
    public class Sync
    {
        private Differences GetDifferences(string dbServerNameSource,
                                               string dbUserNameSource,
                                               string dbPasswordSource,
                                               string dbNameSource,
                                               string dbServerNameDest,
                                               string dbUsernameDest,
                                               string dbPasswordDest,
                                               string dbNameDest)
        {
            Differences differences = null;
            Database db1 = new Database();
            Database db2 = new Database();
            db1.Register(new ConnectionProperties(dbServerNameSource,
                                                  dbNameSource,
                                                  dbUserNameSource,
                                                  dbPasswordSource),
                                                  Options.Default);
            db2.Register(new ConnectionProperties(dbServerNameDest,
                                                  dbNameDest,
                                                  dbUsernameDest,
                                                  dbPasswordDest),
                                                  Options.Default);
            differences = db1.CompareWith(db2, Options.Default);
            foreach (Difference difference in differences)
            {
                difference.Selected = ExcludeObject(difference);
            }
            return differences;
        }
        private ExecutionBlock ScriptDifferences(Differences differences)
        {
            Work work = new Work();
            work.BuildFromDifferences(differences, Options.Default, true);
            ExecutionBlock block = work.ExecutionBlock;
            return block;
        }
        private BlockExecutor SyncDifferences(string dbServerNameDest,
                                      string dbNameDest,
                                      string dbUserNameDest,
                                      string dbPasswordDest,
                                      ExecutionBlock block)
        {
            BlockExecutor executor = new BlockExecutor();
            executor.ExecuteBlock(block,
                                  dbServerNameDest,
                                  dbNameDest,
                                  true,
                                  dbUserNameDest,
                                  dbPasswordDest);
            return executor;
        }
        private static bool ExcludeObject(Difference difference)
        {
            if (difference.DatabaseObjectType != ObjectType.User &&
                difference.DatabaseObjectType != ObjectType.Role &&
                difference.DatabaseObjectType != ObjectType.Queue &&
                difference.DatabaseObjectType != ObjectType.Service &&
                SpecialObjects(difference) == false)
            {
                return true;
            }
            return false;
        }
        private static bool SpecialObjects(Difference difference)
        {
            if ((difference.DatabaseObjectType == ObjectType.Table &&
                 difference.Name.ToLower().StartsWith("[dbo].[aspnet_sql"))
                (difference.DatabaseObjectType == ObjectType.StoredProcedure &&
                 difference.Name.ToLower().StartsWith("[dbo].[aspnet_sql"))
                (difference.DatabaseObjectType == ObjectType.StoredProcedure &&
                 difference.Name.ToLower().StartsWith("[dbo].[sqlquery")))
            {
                return true;
            }
            return false;
        }


        public object SyncDB(string dbServerNameSource,
                           string dbUserNameSource,
                           string dbPasswordSource,
                           string dbNameSource,
                           string dbServerNameDest,
                           string dbUserNameDest,
                           string dbPasswordDest,
                           string dbNameDest)
        {
            object obj = null;
            Differences diff = GetDifferences(dbServerNameSource,
                                         dbUserNameSource,
                                         dbPasswordSource,
                                         dbNameSource,
                                         dbServerNameDest,
                                         dbUserNameDest,
                                         dbPasswordDest,
                                         dbNameDest);
            ExecutionBlock script = ScriptDifferences(diff);

            obj = SyncDifferences(dbServerNameDest,
                            dbNameDest,
                            dbUserNameDest,
                            dbPasswordDest,
                            script);
            return obj;
        }
    }
}