Thursday, January 18, 2007

Building Castles On Quicksand

The projects I am involved in for my "day job" are moving to .NET 2.0 so lately it has been head down converting a huge array of wizards, templates, add-ins and of course the code itself. I also rewrote the core of NCoverExplorer over Christmas as part of a future 1.3.6 release but have had no further time for CC.Net, NCover or various other tools I had been tinkering with contributions for. Once things quieten down a bit at work I will be able to crank things up again.

Moving to .NET 2.0 at work has resulted in various amounts of pain thanks to bugs in the .Net framework, the IDE, the designers and the extensibility library. I've mentioned before I am not a fan of VS.Net 2005 and using it in anger these last few weeks has done nothing but lower my opinion. How some of these bugs or design decisions got through "quality control" is beyond belief. Actually I do have a theory on that but I will save that rant for another day.

The reason for this post however is to blog about my experiences with Sandcastle and generating help documentation for VS.Net 2005.

We have been using NDoc of course for our .NET 1.1 based documentation for VS.Net 2003. Unfortunately as was much documented and debated a while ago, Kevin Downs pulled the plug on this product. Part of the rationale given is Microsoft's announcement of Sandcastle, now up to it's third CTP release (Dec 2006) so I figured it might be mature enough to at least try it on for size.

Wrong, Wrong, Wrong.

If you haven't tried it and have been wondering on the sidelines like me - let me save yourselves hours of pain and frustration by suggesting you continue to wait. It is labelled as CTP but it is closer to the prototype end of the scale than the release candidate one - nowhere near ready for community usage, at least for our requirements.

So what got me so underwhelmed about it?

Well for starters the fact that Microsoft haven't yet included any form of command line or GUI means straight away you are into the world of evaluating and choosing other tools to actually make it work. Yes, I know you can argue choice is a good thing and I don't mean to denegrate in any way the effort people out there have put in to try to make up for this shortcoming by Microsoft. However sometimes you just want only one new moving part to add to your build process.

Next you need to install the VS.Net 2005 SDK to get the right Help2 compiler - it is no longer available as a separate small download like it was for VS.Net 2003 with VSHIK.

Then there is the list of "known issues" - mostly problems with the templates which mean your output is going to be kakked so you need to replace them. So that's usually four separate "things" to be downloaded, installed, configured and documented for your build server before you even try to configure a documentation project - straight away you are aware this is not going to be trivial.

So how do you automate Sandcastle? Well, there is a wiki which indicates at least some of the options contributed by the community - a batch script , an MSBuild script and several GUI options, the most popular being an NDoc lookalike called Sandcastle Help File Builder (SHFB). There are VS.Net add-ins as well although I'm after a tool that works as part of our automated build process - not locking up the IDE for hours thanks.

A dozen-step batch script didn't sound too promising to me, so I tried the MSBuild script by Anders Ljusberg first. Simple to deploy, a few tweaks for paths here and there, ran it and *boom* - first technical issue.

By default Sandcastle wants to connect to a web service on the internet - I believe to generate links to Microsoft content. Well that's just great if you live in the magical fluffy world where proxy servers don't exist, or if they do they take IE settings. Unfortunately where I work we have popup dialogs for authentication - which isn't catered for in any way I know of by the standard <system.net> configuration override trick. The "Start" page in Visual Studio.Net 2005 suffers from the same problem.

I couldn't see an obvious way to turn off that type of online linking using the MSBuild script (perhaps someone can tell me how) so that knocked it out of the equation for now. Incidentally I ran the script on my home PC which has no proxy server and hit a different issue, so it's clearly not the only problem you may face with using it.

Next I moved on to SHFB which does have an option in the GUI to set the external linking to "none". This looked far more promising having both a console and GUI version in NDoc style, some help etc - kudos to Eric Woodruff for his efforts. The price you pay over the MSBuild approach is less flexibility in terms of dynamically assigning assembly lists, version numbers etc since it works off configuration files. No major problem though as it is easy enough to knock up a NAnt or MSBuild task to generate the configuration file dynamically. First priority though is getting the output generated.

And that's where problem #2 came up - performance or rather appalling lack thereof. Generating a .CHM file for our 40 assembly application which used to take an hour under NDoc came in at 5 hours with Sandcastle. WTF? Ok, so leaving it running overnight is going to be the Sandcastle way of doing things. BTW, according to SHFB it took 4 minutes 55 seconds - rather wishful thinking!

There were a few problems with the output so I tweaked some settings (like saying I don't want namespace summaries as I dont want big red warnings everywhere) and ran it for VS.Net 2005 output on just three assemblies. Out popped the help - but still the big red warnings! Double checked my GUI configuration - show warnings are set to false. Hmmmm... Equally bad is that there is a horrid empty blue bar near the top of each page - I think it is expecting some sort of input to go in there but can't cope with it not being present.


So after all the research, downloads, configuration, bugs, trial and error I end up with something that takes hours to render and looks crap. Not quite the result I was after - yes I could try to fix the bugs but that's not what I set out to do. I just need to generate some documentation.

So what to do in the mean-time? I had tried the NDoc2005 project on SourceForge but found it completely unusable with bugs and the developers have clearly lost interest.

I then stumbled across something known as NDoc Alpha which appears to be the last work in progress build of the original NDoc developer Kevin Downs. It all sounds very promising on the web page in terms of support for generics and performance boosts.

I downloaded it and gave it a whirl for a .CHM file,and *boom* - 'access is denied' messages. NDoc Alpha copies your assemblies to a shadow folder so as not to lock the originals. The downside is if those assemblies have the read-only flag set it tries to do a delete from the shadow folder to clean up but doesn't check for the flag first. Now that is a pain - I wanted to build from my ClearCase view release folder which has those attributes set. Thankfully there is an option "UseAssemblyShadowCache" which when set to false disables this feature and hence works around the bug - nice.

Once I hacked that I was able to set NDoc Alpha off in motion again... and *boom* hit a series of big nasty exceptions which stopped documentation being generated. After much fiddling I figured out it was an xml tag that had the wrong attribute that VS.Net does not validate as an error. FYI it was <exception name="..."> instead of <exception cref="...">. In later iterations I also found it erroring where the tag had no attribute at all - i.e. <exception>. It appears that the new CHM documentor in this build of NDoc is far stricter than previous NDoc versions - not a bad thing, just it would be nice to have a more useful error message to figure out what is wrong with the tags.

Incidentally I had discovered that NDoc is spitting out Trace messages as it processes elements - that makes life a bit easier to find out what class it was processing when it failed. Fire up your favourite debug trace viewer such as from SysInternals and you hopefully get a hint as to either the method that failed or at least what once had processed successfully in the class.

Finally with all the error messages cleared NDoc Alpha had a clear runway. Boy is it quick! It took a mere 15 minutes to build the documentation from scratch and generate the .CHM file and a similar amount for the VS.Net 2005 version. Now that's just a complete no-brainer compared to Sandcastle. Sorry guys, you have a long, long way to go.

All that remains now is knocking up a new <ndoc2> NAnt/MSBuild task so I can generate the configuration file on the fly. Shouldn't be a major deal, and I will include the <ndoc2> task I write in NCoverExplorer.Extras.zip when I'm done for anyone interested.

The not inconsiderable negative of NDoc Alpha is that it is an end of line product - no source code is available to fix the bugs or anyone to offer support. It looks like Kevin has the last laugh on this for a while longer - his product is indeed still light years ahead of Sandcastle but the community are not going to be able to advance it further. That's frankly a damn shame as this is undoubtedly the best version of NDoc yet - after all the years of effort to let the source code waste away in the cupboard rather than live on in the community by others is a major downer... forgive the pun.

In the mid-long term then undoubtedly provided Microsoft deliver on the hype then they will be the product of choice - particularly for people not looking to use a commercial offering. For now though Sandcastle is a major thumbs down for me.

BTW - does anyone actually like the new "language filter" that is a part of the VS.Net 2005 help? You only work in one language but have to do half a dozen clicks to just see the C# examples by unselecting everything else first? Then if you get a page which doesn't have the language you are after, you go through another clickfest to see if there was an example in another language instead? If only Microsoft would make it a persisted option - setting a default language with an ordered list of alternatives if none of your defaults exist, how hard is that?

If anyone reading this feels I'm being too harsh or you wish to correct me please feel free to slap me round in the comments. My disclaimer is I'm under pressure to get stuff out the door so my "dive in and fix it" threshold is much lower than normal - using the current versions of Sandcastle and the related tools is not for people in a hurry from my experience.

Filed in:

13 Comments:

At January 19, 2007 2:45 PM, Anonymous Anonymous said...

My god! A CTP product with bugs, missing features and bad performance?!

Someone call the cops!

:-)

 
At January 19, 2007 3:12 PM, Blogger Marcos Meli said...

Very good POST !! =)

Completely the same problem for me, I can tag my Sandclastle experience as a TIME LOSS !!

I was trying to use it for www.filehelpers.com but, by the time, I´m still with NDoc, no generics, but I complement it with the Docs =)

About the "language filter" is a good thing if well implemented !!!

How they can forget a "Select all" "Select None" link !!! for the love of good is so so simple, and like the source is propietary, maybe we never get them, the second bad thing is the ONLINE MSDN2 dont remember right my Language selection !!

very very bad M$ you lost another 2 cents with that.

Cheers
Marcos

 
At January 19, 2007 4:29 PM, Anonymous AW said...

Cut your losses and use this:-
http://www.innovasys.com/products/dx5/overview.asp

 
At January 19, 2007 5:17 PM, Anonymous Travis Illig said...

I've been working with Sandcastle for a while now to integrate emulation of it into my CR_Documentor doc preview tool and I couldn't agree more - Sandcastle has a long way to go to even become usable.

I actually found the biggest shortcomings in the tag support in Sandcastle. Tags I use all the time that work fine in NDoc aren't supported (or are implemented incorrectly) in Sandcastle. When you ask for Sandcastle to support these tags - maybe they're not the documented standard on MSDN, but they are de facto standard, a concept I'm sure Microsoft is familiar with from its IE experiences - you get pointed to a wiki entry explaining how to implement them yourself in a custom fashion. Not super helpful.

I won't even go into the pain of running several different commands just to generate the docs. Transform this bit into this format, then transform that into this other format, then integrate this bit over here into it, then compile it... I shouldn't have to know about that. I surely didn't with NDoc.

Ah, well. Unless someone wants to take over NDoc, there may be no other choice.

 
At January 19, 2007 7:47 PM, Anonymous Anonymous said...

We're using SandCastle (integrated into our CCNet/NAnt build process) for a while now, and although it still shows some problems (the language filter thing is annoying), I think it's a feasible option to generate .NET 2.0 class library docs.

And yep, performance is close to unacceptable, the web service calls take forever. But we can live with that for the moment, since we're generating the SDK docs only in the nightly release builds.

Karl Traunmueller
CTO, Objectbay Software

 
At January 20, 2007 2:04 PM, Blogger kiwidude said...

Thanks for the comments guys.

Anonymous - perhaps your comment is tongue in cheek? Yes it is CTP but I have serious concerns about its design. After six months publicly of development (how many before Microsoft announced it?), at least 3 iterations by a company with all their resources who create the standard and have been generating MSDN documentation for years - this is the best they could come up with? Sandcastle has some fundamental bad smells about it - and unfortunately with little non-commercial alternatives people are being forced to try it.

Marcos/Travis - sounds like you know the pain too then. All those wasted hours - arggghhhh!

Speaking of commercial - thanks AW for suggesting an alternative. Unfortunately in large corporates like where I work it takes 6+ months to get a new tool onto our desktop - it has to be budgeted for the year before, multiple approval cycles, committee meetings, legal, vendor approval, purchasing, packaging etc. An absolute nightmare (whether it costs $30 or $1000 is irrelevant really) so we unofficially tend to try to use tools which don't require that process. Unless you are a software vendor, generating MSDN style documentation is pretty low down the list of priorities for having a tool already on the approval list. Then when Microsoft "announce" a free tool that usually throws a spanner in the works of getting a purchase of something similar. Although hopefully enough people realise by now that Microsoft very rarely imho deliver anything of real quality in their first release or two to compete with alternatives. VSTS testing/build process is a prime recent example, with VS.Net 2005 refactorings another.

Karl - good to hear from someone who is using it and stuck with it. I've seen a number of posts about people announcing this or that tool for trying to work with it, but few about real world experiences. Since I posted I have found an excellent post by Travis from back in Sept (follow the link to my post of his). Three months later and the situation has not improved in any way - still no docs, no front end and still the same clunky architecture.

 
At January 22, 2007 1:43 AM, Blogger si said...

Hi, I agree with some of your comments, but perhaps I've been lucky, or are more forgiving, as it hasn't been that bad an experience driving Sandcastle from NAnt and CruiseControl.

To answer of your questions, regarding the need to resolve reference links, here's what we do :

...
<!-- Sandcastle supports 4 different link type options.
1. none - Results in text with no active link
2. local - Results in links within project using <a href> tag
3. index - ms-help style links for HxS
4. msdn - links to Framework topics in MSDN -->
<property name="resolveReferenceLinksComponent" value="msdn" overwrite="false" />
...
<!-- Modify sandcastle.config to reference absolute path and our assemblies xml -->
<copy file="${sandcastle.dir}\Presentation\${style}\Configuration\sandcastle.config"
tofile="${doc.temp.dir}\sandcastle.config">
<filterchain>
<replacestring from=""..\..\" to=""${sandcastle.dir}\" />
<replacestring from=""..\" to=""${sandcastle.dir}\Examples\" />
<replacestring from="comments.xml" to="${doc.temp.dir}\${assemblies}.xml" />
<replacestring from="type="msdn"" to="type="${resolveReferenceLinksComponent}"" />
</filterchain>
</copy>
...

If you're interested in the full nant build script, let me know (sshnug.si at gmail), and i'll email you the script, but it will probably need some modification.

There are certainly some tricks, like merging a separate xml file so you can get namespace documentation, and those guid html docs are horrid (but fixable). But the only thing we're really missing is the ability to easily add non-class based documentation, like Microsoft do with their MSDN docs. e.g. an overview of a class.

 
At January 25, 2007 5:25 PM, Blogger Thomas Hansen said...

Thanx, you've just saved me from days of Sandcastle "punding my head into the wall" hours...
http://ajaxwidgets.com

 
At January 29, 2007 4:11 PM, Anonymous Warwick said...

Great article Grant!

Have you (or anyone else out there) tried doc-o-matic? http://www.doc-o-matic.com

We've been scratching our heads looking for an alternative, and this was all we could find. Its quite pricey though, but it does seem to have Help2 generation, VS2005 integration etc etc.

Cheers

Warwick

 
At February 08, 2007 12:29 AM, Anonymous Gareth Hayter said...

I have also been thoroughly discouraged by Sandcastle. I have been looking at various commercial tools. The one I like best so far is VSDocMan, because it is integrated into VS 2005, and has round-trip support. Very nice. Here are links to others I have looked at, as well as a wikipedia page on code documentors:

VSDocMan: http://www.helixoft.com/vsdocman/overview.html

Doc-o-matic:
http://www.doc-o-matic.com/

TwinText:
http://www.ptlogica.com/TwinText/overview.htm

Help Generator:
http://www.helpgenerator.com/

Wikipedia listing:
http://en.wikipedia.org/wiki/Comparison_of_documentation_generators

 
At February 14, 2007 4:08 PM, Blogger Gunnlaugur said...

"I also rewrote the core of NCoverExplorer over Christmas as part of a future 1.3.6 release"

Just as a nitpicky sidenote, is "rewriting the core" really something one does in a subminor-version update like 1.3.5 -> 1.3.6? To me, such a small version number increment implies a bugfixes-only release.

 
At February 14, 2007 7:27 PM, Blogger kiwidude said...

Gunnlaugur,

A fair point. I mentioned 1.3.6 as a way of indicating to people that there would be a new version. In actual fact the code has been branched and the "core rewrite" will indeed be in the 1.4 version.

It's all fairly moot at the moment as I haven't got time to work on any version - hopefully in a week or two...

Grant.

 
At March 15, 2007 1:48 PM, Anonymous Anonymous said...

I see they fixed 100s of issues in the new March preview.

Having seen these comments, has anyone dared to install it?

Trevor

 

Post a Comment

Links to this post:

Create a Link

<< Home