Vampire Basic

Running Mercurial on Windows Without CGI

2010-06-18T09:45:00.001-05:00

In the past, the only real way to run python apps in IIS has been through CGI. FastCGI doesn't work because of an implementation flaw in Python 2.x script libraries that basically prevents FastCGI from functioning on Windows (supposedly this is fixed in Python 3.x). ISAPI wasn't really an option, either, since there was not really a good enough implementation of the WSGI server interface standard for IIS.

By the way, for those of you who haven't been keeping up with the inner workings of Python (that would be most of us Windows folks), WSGI (Web Server Gateway Interface) is a Python standard defined in PEP 333 (PEP is an acronym for Python Enhancement Proposals). It details how Python web applications/Python web frameworks should interface with web servers.

Things have change a little in the months since I wrote my original post on running Mercurial on Windows. Someone who has a great deal of dedication has been working on a WSGI interface for IIS since before my last post, and has now gotten it to a point where it works marvelously with Mercurial. Thanks to Jason R. Coombs and his team for all their hard work on the isapi-wsgi project.

On with the show

Installing Mercurial as a server using isapi-wsgi is SO much easier than fiddling with CGI that it was actually a joy. These instructions assume Mercurial 1.5.1. Keep in mind that since you don't have anything on Windows as cool as the FreeBSD Ports and Packages Collection to build stuff and manage dependencies for you that you will have to make sure you match the version of Mercurial to the correct version of Python yourself. IIS needs to be already installed and functioning.

Install the Python 2.5.6 binary installer package.
Install the latest of Mark Hammond's indispensable Python for Windows package, for Python 2.6 .
Install the binary package for isapi-wsgi. 0.4.2 is the one that you want, since it now supports SSL properly. isapi_wsgi.py will be installed in the 'site-packages' directory of your Python installation...assuming you take the defaults. You should take the defaults.
Go here and follow the instructions in the Testing section to make sure you didn't mess it up.
Go to Selenic and download the source code for Mercurial. Its in a tarball, so you'll probably want to install 7-Zip at this point, because Microsoft obstinately refuses to include tar and gzip support directly in the Windows Explorer...and really why should they since 7-Zip is free, and does exactly that.
Once you've extracted this thing somewhere (or not, you can navigate the archive in 7-Zip and drag-drop the file we need out), find the contrib\win32 directory. Copy hgwebdir_wsgi.py to a directory from which you want to serve Mercurial. I chose C:\inetpub\wwwroot\hg. It doesn't have to be in inetpub, you could use a virtual directory. Don't put your Mercurial repositories in this directory. You'll confuse Mercurial.
On IIS 6 and above, create a new App pool for your Mercurial directory if your IIS installation is doing ANYTHING other than Mercurial. Make sure you make your Mercurial directory into a Application in IIS, and ensure the Application is assigned to the App pool you created for Mercurial. In IIS 5.1, you may want to set the Application Isolation as High "(Isolated)". I mean, the Python interpreter is native code. It could crash and take your entire IIS process down with it. You probably don't want that.
Edit your copy of hgwebdir_wsgi.py. Make the top of it look kinda like this:
```
# Configuration file location
hgweb_config = r'c:\inetpub\wwwroot\hg\hgweb.config'

# Global settings for IIS path translation
path_strip = 0   # Strip this many path elements off (when using url rewrite)
path_prefix = 1  # This many path elements are prefixes (depends on the
                 # virtual path of the IIS application).
```
hgweb_config is pretty simple, it tells the script where to find its config file. path_prefix should be 0 if hgwebdir_wsgi.py is in the root of the web, 1 if its one directory down (server/hg), 2 if you get to it by server/directory/hg... you should get the idea. I don't know what path_strip does, but it sounds risque, so I didn't mess with it.
Run the hgwebdir_wsgi.py script
```
python hgwebdir_wsgi.py
```
to make a shim dll for Mercurial. All this really does is copy a file from the Python Win32 Extensions package to where the script resides and names it appropriately. Which is nice because you don't have to figure it out for yourself.
Create your hgweb.config file. This is all mine has in it.
```
[paths]
/ = c:\repos\*
```
If you read my last post, then yes, there is just one star this time.
Create an ISAPI script mapping to the shim dll that the script generated. In IIS 5, you just create a script mapping to the dll using .* as the extension (clear the check box that says "Check that files exist", or IIS will generate hate messages). In IIS 6 and above, you will want to create a wildcard script map to that shim dll.
Create a test repository in c:\repos\ (or where ever). If it shows up in the list on http://server/hg, have you a cold one. Good job. If it doesn't, you may have messed up your hgweb.config, maybe your permissions on c:\repos\. Remember, the identity that IIS (or the IIS worker pool) is running under will need full control access to your repository directory. If you don't see anything at all (or 401's and 404's and bears, oh my!), you may have messed up your script map in IIS or the editing the hgwebdir_isapi.py script.

SSL and web security both worked well through IIS. One thing to keep in mind is that the Hg client doesn't care a thing about Windows Integrated Security, so if you want to secure your repos, you'll have to use SSL+Basic Authentication.

If you find yourself having to diagnose your setup, iisreset is your friend. Because you're loading things into the IIS process space, and IIS isn't very likely to reload things just because you changed them, you may find that you have to restart it when you make certain configuration changes.

I found setting up Mercurial with isapi-wsgi to be easier than with CGI. It seems snappier, and less fussy about certain operations that I had trouble with in CGI. I imagine if FastCGI were to ever work with Python on Windows, it would actually perform better than ISAPI, due to the nature of threading within the Python interpreter, but for now, we already have something better than CGI, so that's what I'll stick with.

Issues I've seen

On a 64 bit Windows 2008 R2 server, I ran into a bit of trouble with Mercurial not being able to find its binary bits. See Mercurial has some modules in it that were written in C, because they're really hard core and mathy and they needed some extra speed to make Mercurial as fast as it really should be. In order to fix the "unable to locate module" business, I downloaded the Mercurial source, and lo, the authors had written pure Python versions of the C modules that were giving me grief. I tossed the pure python modules on the PYTHONPATH, and everything worked just fine.

I'm going to research this particular issue again, because I've learned a little more about how Python loads C code. It sure would be nice to know how to get Mercurial running at full tilt in IIS 7.5.

How to debug an ISAPI Extension with Visual C++ 2010 Express on Windows XP

2010-04-20T11:06:00.001-05:00

I had a great deal of trouble figuring out how to debug an ISAPI extension I'm writing, and unfortunately, most of the information on the internet about ISAPI has gotten a little stale since the advent of .Net for web development.

On Windows, native code is moving further into the realm of witchcraft. Because witchcraft is scary, the Visual Studio team hid all the powerful witchery in 2010 C++ Express Edition. To turn all the really useful stuff back on, you have to go to Tools > Settings and select "Advanced Settings". Now you can get at the "Attach to process..." feature.

That wasn't what hung me up. I followed the instructions in the MSDN library, which basically say "Attach debugger to Inetinfo.exe, enjoy." But I got no joy. I got "The breakpoint will not currently be hit. No symbols have been loaded for this document." in the interactive debugger.

So, I enabled symbol download in Visual C++ by navigating through Debug > Options and Settings... and clicked on Symbols in the tree view on the left. Check the box next to Microsoft Symbol Servers, created a folder to hold whatever it downloads and put that path in the Cache symbols in this directory: box, and I was ready to go. Except it didn't help me out.

I loaded the Windows SDK, I loaded the Debugging Tools for Windows (which are inconveniently in a 650mb iso with the Windows DDK), I installed depends.exe (which apparently does not come with the Windows SDK anymore), I installed SysInternals ProcMon.exe. Of all the hundreds of megabytes of junk I downloaded and installed, none of the tools told me what I needed to know.

I'll save you the effort of installing all that stuff and just tell you that you need to put your IIS Applications into Application Protection: Low(IIS Process). If you don't, the ISAPI dll you are trying to debug gets loaded into some dllhost.exe process. If you leave the IIS Application set to Medium(Pooled) or High(Isolated), you will have to attach your debugger to one of the dllhost.exe processes. It is harder to figure out which dllhost.exe to attach the debugger to, so set it Application Protection to Low, THEN follow the instructions on the MSDN Library.

So, long story short, I was attaching the debugger to the wrong process. And you can't debug code that isn't loaded into memory. It seems a shame that that is the case, because if it weren't, then native code really would be witchcraft.

Running Mercurial on Windows

2009-06-07T10:53:00.035-05:00

Mercurial, if you haven't played with it yet, is one of the easiest, fastest source control mechanisms to hit Windows in recent years. If you have heard of Mercurial (or Hg, as some folks will type it, the symbol on the periodic table of elements for mercury), you may think I misspoke with that "Windows" bit, since only those long-haired open source hippie Linux people use that freebie distributed version control. Not true! As a matter of fact, for your own personal projects, Hg is probably one of the easiest, fasted version control mechanisms you can use for local-only development.

Getting started

The basic reference material you will need for Hg is available in the Mercurial wiki at http://www.selenic.com/mercurial/wiki/. Just so you don't have to read it, I'll continue below.

On your local Windows box, you will want to install TortoiseHg. If anybody has ever used the fine TortoiseSVN on Windows, you will be immediately familiar with the direct Windows Explorer integration with source control. Some of the more recent versions of TortoiseHg (0.7.6 is the latest as of this writing) include a full installation of the command-line version of Mercurial, saving Windows uses the hassle of finding and running another installer. You will eventually come up with a need for the command-line version of Hg, and you will want to become familiar with its simple commands anyway.

For Visual Studio users (Standard edition and above, sorry Express edition users. Out in the cold as usual.), you can install VisualHg to get that sweet, sweet development environment integration that the users of Visual Source Safe and TFS currently enjoy in their development experience. Once installed, you can enable the source control integration within Visual Studio by going to Tools->Options, then choosing "Source Control" in the tree view, and changing the "Current source control plug-in:" combo box.

As a side note, if you're still using VSS...please stop. SVN and Mercurial are, at the very least free, and are far less likely to corrupt your source repository. Also, if you aren't paying for each and every seat license of VSS (even if you put it on a server and share it), you're stealing from MS at the rate of about $390 a seat. Don't place your business and your code at risk by using VSS.

Beyond local development

Despite what some other ill-informed (yet admittedly old) blogs have implied, Mercurial can be hosted in IIS on Windows. Nobody's set out step-by-step instructions all in one place, but that is about to change.

For all versions of Windows

Install Mercurial 1.2.1 on the server. The version that comes with TortoiseHg is not sufficient, it will not drive the CGI application.

Install Python 2.5.4 on your server. Mercurial 1.2.1 components are compiled against that version of Python. Any other version of Python will cause a "Bad Magic Number" error on the server.

Create a directory somewhere to hold the Mercurial source tree. Open a cmd window and navigate to whichever directory you want this source tree to be contained within, then type in
```
hg clone http://selenic.com/repo/hg
```
(robbed from Stack Overflow). This command will create a directory named hg, then download all the code in the Mercurial source repository into the hg directory. What we're after here is is the hgwebdir.cgi file, which is...a Common Gateway Interface app. Duh. And I'll bet you thought that cgi was dead.

Create a directory on the file system (separate from any other web app files you have on your web server) to map in as a Virtual Directory in IIS. I used C:\inetpub\hgcgi\. Make sure this directory has read access in NTFS for the service account IIS (or your IIS app pool for Vista, XP x86-64, Server 03 and Server 08) is running under.

Copy the hgwebdir.cgi file into your virtual-directory-to-be.

You also want to extract the contents of the Library.zip file contained in your Mercurial installation directory to your virtual-directory-to-be. If you installed TortoiseHg 0.7.6, you will find this within the TortoiseHg installation directory.

Copy the entire "templates" directory from the Mercurial installation directory to a "templates" directory within the virtual-directory-to-be.

Create a file named hgweb.config. This is absolutely nothing like an Asp.Net web.config. Its more like an ini file from the days of old. And you have to actually add text to it. You want something that looks like this:
```
[paths]
MySourceCode = C:\Repositories\**
[web]
style = monoblue
```
The [paths] segment is in the form = local path. You can list out repository locations manually, or you can do as above, and have it take all of the repos in a location. And yes, the ** is correct. Most of the other documentation around the web is unix-style fs specific, and suggest using the [collections] section as opposed to the [paths] section, but don't be surprised when it doesn't work on Windows. I never had any luck with using the [collections] section.
And yeah, you want to change the style. Because you can.

Once you get your repos to show up in the Mercurial cgi script, you can go into the repo's .hg directory, find the hgrc file therein and configure it. The syntax is the same as hgweb.config, and it allows you to set settings for the repository itself. Such as the style. Because you can. Details on the config file are here.

That's everything you must do that is common to all versions of Windows.

IIS 7 (Windows Vista, Server 08)

Just like everything else that is related to Vista, Microsoft changed the whole UI for the IIS Manager. This is good, because IIS Manager (just like the Component Services snap-in) in Server 2003 is a fussy, buggy, expletive-inducing wreck. IIS 7 does give you the ability to install Asp.Net "filters" (called a "Managed Handler" in IIS Parlance, called an HttpHandler in Asp.Net parlance) directly into IIS. Kinda cool really, since writing ISAPI filters requires that icky native code that seems to bamboozle so many people. Maybe someone will write a nice Managed Handler for Mercurial in the future, but for now, we have to use CGI.

Install IIS. In Vista, its Control Panel->Programs and Features in "Classic View" or Control Panel->Programs in the new Vista view. Click on "Turn Windows features on or off." Expand "Internet Information Services", then select "Web Managment Tools" to install IIS Management Console. Select "World Wide Web Services", then expand "World Wide Web Services", then expand "Application Development Features" and make sure "CGI" is selected.

Add your virtual directory for Mercurial. That's the one that has hgwebdir.cgi in it. Right click a web site (I just mapped it into the "Default Web Site", but any will do.), and choose "Add application" You could "Add virtual directory" and then upgrade the virtual directory to an application, but the virtual directory mapping has to be an "application." In the dialog, I set the alias to hg and the Physical path to "C:\inetpub\hgcgi". There is a nice "Test settings..." button here that will help you determine if your application directory is configured property. I never did get my authorization warning to go away, even though the CGI script would run.

Map in the Python interpreter as a script engine. Click on your new application/virtual directory. In the center pane, double-click "Handler Mappings." Normally, if you wanted to execute Python scripts on the server, you would map the interpreter to *.py. But we're running a cgi app through the python interpreter. Click on "Add Script Map..." and put in *.cgi for the Request Path. Put
```
C:\Python25\python.exe -u %s %s
```
in for Executable. Give your mapping a name. I used Python 2.5. The %s %s mess above are substitution symbols (like printf(), or string.Format()) for arguments passed to the python interpreter that tell it what script it is executing on behalf of IIS. After that, click the "Request Restrictions..." button, and choose the "Access" tab. Pick the "Execute" option, so you can "Execute" the cgi application. Click OK to save. A dialog will pop up that says "Do you want to enable this ISAPI application?", which is a bit bizarre since you are enabling a CGI application, but you should say Yes.

Enable the script engine for your application/virtual directory. Right click your new "Python 2.5" handler mapping, and select "Edit feature permissions...", and make sure all of the checkboxes there are checked.

Windows Server 2003/XP

Buy Server 2008. I hear that if you get it on subscription, you get free upgrades. If you can't talk your boss into paying for the next scheduled maintenance on Balmer's V12 Vantage (and if he doesn't really have one, he probably should), make sure IIS is installed on the Server. Control Panel->Add or Remove Programs, then Add/Remove Windows Components. Select Application Server, and you should be set.

Add your virtual directory for Mercurial. That's the one that has hgwebdir.cgi in it. Right click a web site (I just mapped it into the "Default Web Site", but any will do), and choose New->Virtual Directory. In the dialog, I set the alias to hg and the Physical path to "C:\inetpub\hgcgi". By default, in Server 03, virtual directories are mapped in as "Web Applications," but if it isn't set up that way, you can right click the virtual directory, and click the "Home Directory" tab. On the bottom of that tab, in the "Application settings" section, there should be a button labeled "Add." Once you click it, the virtual directory becomes a "Web Application." Also, while you're there, make sure the "Execute permissions:" drop down says "Execute."

Map python in as the cgi script handler. Do this by right clicking the virtual directory, then selecting the "Home Directory" tab. In the "Application settings" section, click the "Configuration" button. On the "Mappings" tab, click the "Add..." button. For "Executable", put in the following:
```
C:\Python25\python.exe -u "%s %s"
```
And make sure you put the double quotes in exactly as shown. This is because IIS Manager is and won't take it without them. In "Extension", put ".cgi". Clear the checkbox that says "Verify that file exists", set the checkbox that says "Script engine."

Enable Python as Web Service Extension (Server 03/XP x64). In IIS Manager, right-click the Web Service Extension to "Add a new Web service extension...". You can name the extension anything you want, I chose "Python 2.5". Click the "Add..." button. In "Path to file", put C:\Python25\python.exe -u "%s %s". It must exactly match the ".cgi" mapping on your virtual directory, or it won't work. Check "Set extension status to Allowed".

Point your browser to http://server/HgVDir/hgwebdir.cgi/ and enjoy. You should now be able to use Mercurial against that virtual directory. To secure your source control repository, you should set up SSL on your server, and mark your hg virtual directory to require SSL. Then set the virtual directory to use Basic security, as this is what the hg client tools support. Basic-over-SSL is pretty secure, since the SSL encrypted connection is established well before the Basic security credential exchange takes place.

To create new repos on the server, you'll want to clone them to the server from your workstation machine. The easiest way to do this is to hg serve from your workstation, then log onto the server and pull your new repo into the right place on the server from your workstation. After that, go into .hg in your server-side repo and configure your hgrc file. Once that is complete, you can synchronize your server repo with your workstation repo any time you wish.

And just in case you're wondering, the -u switch on the python interpreter places the interpreter in "unbuffered" mode, so results are sent to IIS immediately with no delay. Supposedly, for CGI, it improves performance.

Give hg a shot. Its effective and I've had quite a bit of success with it. Maybe you'll like it, too.

Connection per request with Ado.Net

2009-05-30T16:05:00.007-05:00

Say you have a service layer component that has reference to a DAL component. Say that the service layer calls two different methods on the DAL component, and that those methods need to be bound in one single transaction. How do you achieve this within the confines of a web application?

Since transactions are the domain of the service layer (business logic components...whatever), this should be pretty easy, right? Just start a transaction one way or another, call your methods, commit/rollback, and move right along. The devil is really in the details on this, however.

For the sake of this discussion, pretend that this code exists in the system:


'Highly Contrived Example
Public Class Stuff
    Dim _Id As Long
    Public Property Id As Long
        Get()
            Return _Id
        End Get
        Set(value As Long)
            _Id = value
        End Set
    End Property

    Dim _Thingy As Thingy
    Public Property Thingy As Thingy
        Get()
            Return _Thingy
        End Get
        Set(value as Thingy)
           _Thingy = value
        End Set
    End Property
End Class

Public Class Thingy
   ...Whatever goes here
End Class

Public Class DaoBase
    Dim _Connection As IDbConnection
    Public Property Connection As IDbConnection
        Get()
            Return _Connection
        End Get
        Set(Connection As IDbConnection)
            _Connection = Connection
        End Set
    End Property
End Class

'This is a terribly designed component
Public Interface IDAO
    Function SaveStuff(arg As Stuff) As Stuff
    Sub UpdateThingyOnStuff(arg As Stuff)
End Interface

Public Class ServiceComponent
    Dim _DAO As IDAO
    Public Property DAO As IDAO
        Get()
            Return _DAO
        End Get
        Set(value As IDAO)
            _DAO = value
        End Set
    End Property

    Public Sub DoSomethingCool(ByVal someStuff As Stuff)
        Using txn As New System.Transactions.TransactionScope
            someStuff = DAO.SaveStuff(someStuff)
            DAO.UpdateThingyOnStuff(someStuff)

            txn.Complete
        End Using
    End Sub
End Class

I will be discussing the implementation of the IDAO interface.

In a web app, you don't really have any state between requests, except that which you can stuff into Session or stuff into a cookie. You can't really place a database connection object into Session. Well, you can, but when you have to switch from in-memory Session to, say SQL Session state, you're screwed. Nevermind having to figure out what to do about fixing the connection if it breaks down on you. And it scales like crap, too, because sessions may take a long time to become "abandoned" as far a the server is concerned, so you end up with database connections being held open that are idle.

You surely can't cram a connection object into a cookie, so that means that you have to open and close database connections within the scope of a single web request.

The wrong way

AKA, the way I used to do it.


Imports System.Data.SqlClient

Public Class DAO
    Implements IDAO

    Public Function SaveStuff(arg As Stuff) As Stuff
        Using conn = New SqlConnection
            conn.Open

            Dim cmd = new SqlCommand
            'Setup the command object here

            cmd.ExecuteNonQuery
        End Using
    End Function

    Public Sub UpdateThingyOnStuff(arg As Stuff)
        Using conn = New SqlConnection
            conn.Open

            Dim cmd = new SqlCommand
            'Setup the command object here

            cmd.ExecuteNonQuery
        End Using
    End Sub
End Class

The problem with this DAL is that the connection doesn't stay open during the transaction. Using System.Transactions as I have done here, the ambient transaction elevates to a Distributed Transaction when you open the new connection in the second DAL call. This means you have to actually enable and configure the MSDTC, and it degrades performance.

If you use ADO.Net transactions instead of System.Transactions, you still have the same problem, except that you now have to keep up with a transaction object. With ADO.Net transactions, you could be creative and retain the connection when you pass in a transaction object, but you still have to come up with some strategy to close the connection when your transaction ends. You could place a Close() method on your DAL components that your service layer components could call, but it just seems so...hacky.

The right way

Open your database connection at the beginning of your web request and close it at the end.


...inside global.asax

Protected Sub Application_BeginRequest(sender As Object, e As EventArgs)
    'Database vendor-agnostic code.
    Dim connectionString = System.Configuration.ConfigurationManager.ConnectionStrings("sampleDbConnString")
    Dim provider = System.Data.Common.DbProviderFactories.GetProvider(connectionString.Provider)
    Dim connection = provider.CreateConnection

    connection.Open

    'The Context.Items collection is a per-request information store.
    Context.Items("DbConnection") = connection
End Sub


Protected Sub Application_EndRequest(sender As Object, e As EventArgs)
    Dim connection = DirectCast(Context.Items("DbConnection"), System.Data.IDbConnection)

    If connection IsNot Nothing Then connection.Close
End Sub

Doing the above leaves only the question of how to get reference to the connection into the DAL components. Since you end up having to construct your DAL components at some point anyway (say, Page_Load), you could simply fetch the connection object off the Context.Items collection and place it on the DAL component.


...in Page_Load somewhere...
me.Service = New ServiceComponent
me.Service.DAO = New DAO
me.Service.DAO.Connection = Context.Items("DbConnection")

This gets you out of DTC elevation with System.Transactions, and you don't have to worry about opening or closing database connections anymore. With Ado.Net transactions, you do still have to fiddle with transaction enlistment, so you still have to pass in transaction objects (good reason to not use it, System.Transactions is easier to use).

For most common web apps, using connection-per-request and System.Transactions improves code quality and reduces the amount of code you have to write. That's good for everybody.

Those wacky Java guys, and their Eclipse

2009-05-11T12:43:00.010-05:00

How much did your company spend on Visual Studio.Net, just so you could write Asp.Net web applications? How much more did they spend if they decided to get the fancy "Team Systems" edition (which they'll pay an annual subscription renewal for next year)?

Then, you have to buy Resharper, because writing code without it is 35%* harder. I would say that without Resharper, code is 50%* harder, except that you have to keep turning it off when you go to edit markup because of performance issues (boo, JetBrains, boo. I know you can do better, I've seen TeamCity).

Now, why mention all of this? Because Eclipse does everything Resharper + Visual Studio.Net does, plus it makes Java not be (complete) poo. And its free. And it makes a great desert topping.

I've also been impressed with Spring, Hibernate, and Struts 2. Struts integrates Spring, allowing Struts to just use Spring to obtain objects, all behind the scenes without writing any code. Struts also integrates Dojo, a Javascript framework similar in function to JQuery. The Struts-Dojo integration reminds me of what Asp.Net Ajax was trying to do with UpdatePanels, only it actually works. Yeah, I said it.

To top it off, Spring is (unsurprisingly) easier to use in the Java world, as you don't have to specify assembly location or version to the Spring runtime...what with Java not having assemblies and all.

Hibernate now uses annotation (attributes in C#) based mapping, doing away with xml mapping files, and Eclipse allows you to generate your entities right off the database. Furthermore, Struts (God bless them), is not strictly typed, so there is no type casting done in the jsp markup, and no writing weird little helper functions to get around C# and Asp.Net's seeming lack of cohesion.

You know, stuff like this:


<asp:label runat="server" ID="SomesuchImportantLabel" value='<%# Convert.ToString(((DataEntity) container.DataItem).DomainDataItem.SomeDate) %>'/gt;

Would look like this in Struts:


<s:property value="DataEntity.SomeDate" />

Hey, Asp.Net and C# teams. You guys ever talk to each other about that whole Web Forms situation? It kinda sucks. Sorry to be rude about it, but VB.Net works seamlessly in Web Forms and with C# you have to do all that clunky casting. It'd be nice to leverage that slick "dynamic" behavior in the new C# language in Web Forms by default. It would put an end to all of that carpel-inducing cast-ification. Granted, your MVC stuff is MUCH better, and more comparable to Struts 2...I digress.

What's not to love? Err...well I did run into a few problems.

Eclipse's jsp editor has an irritating bug in it where it eventually just quits providing code completion, and pops up a dialog every time you press a key, telling all about this marvelous StackOverflowException that it discovered. Closing and reopening the jsp file fixes the problem, but it is disruptive, none-the-less.

Since Struts 2 isn't strictly typed, when you type in something incorrectly, it will just politely ignore it. This bit me mostly because I didn't expect it, given Asp.Net's ruler-to-the-hand response to typing things incorrectly.

Another thing that struck me was how easy it was to configure a JSP application. The web.xml file is tiny, and once you set it up, you can ignore it forever more. This as contrasted to how unbelievably annoying it was to try to find all of the dependencies for the technologies I was using. ClassNotFoundExceptions sent me to the internet more than once trying to scrounge up which jar file a particular class was hidden.

The last thing that slowed me down was not JavaEE, Tomcat, Eclipse, Struts, or JSP. It was the database server. Since I like making things harder than the should be, I decided to use a database server with which I have limited familiarity, PostgeSQL 8.3. Very good database. Except, the damned thing doesn't care for upper-cased letters (as I've noted before in another post). Placing escaped double-quotes into the JPA mapping annotations and explicitly defining catalog, schema, and name is a must if you use upper case characters and PostgreSQL, an issue I would never have encountered with MS SQL Server (it only gets pissy about spaces in object names).

The JPA development aspect in Eclipse isn't really astute enough to recognize the double-quote trick with PostgreSQL, and it gets upset when you put them in ('I don't know what this "Monkey" table is, you should have a Monkey table instead, I saw it on the DB.').

Despite the learning curve involved in switching from the .Net realm to Java, and indeed, most of my issues fall into the learning curve category, I wouldn't hesitate to suggest a survey mission into that world to any company trying to write a brand-new application or perform a serious (Web Forms to ASP.Net MVC) upgrade. Especially since Asp.Net MVC and Struts 2 are very competitive, feature-wise.

*Figures not to scale.

ASP.Net Ajax in Mono

2009-04-11T10:22:00.004-05:00

In a recent discussion with a friend of mine, it was brought to my attention that one of the additions in Asp.Net Ajax (the features located in System.Web.Extensions.dll) was the ability for asmx web services to emit lean-and-sexy JSON instead of tubby-and-pimply SOAP.

This is an exceptionally cool feature, since it effectively gets you out of having to deal with the horrifically complicated configuration involved in setting up WCF, just to get JSON to your client script.

Since web applications are becoming more client-script/web service oriented these days, I was curious at how well these kinds of ASP.Net applications would port to Mono. After all, with the economy being what it is, some companies are becoming interested in ways to continue to innovate while saving some money, and Mono runs ASP.Net on Linux.

In order to get started quickly, I got a copy of VMWare Player (which is free), and downloaded the VMWare image from the Mono project's web site for Mono 2.4. When I started the image, I discovered that everything I needed to start experimenting was ready to go out of the box. There were some samples, and a copy of the new version of MonoDevelop, just waiting for me.

The first thing I did was to create a new ASP.Net web project in C#, and create an asmx web service that would return a complex type of some sort, and then use JQuery on the Default.aspx to call the web service. I used the JSON2.js serializer on the client side to verify that the output by just deserializing it.

The initial results were a disaster. All I could get from Mono's asmx web service was XML output, despite putting the ScriptMethodAttribute on my web service method. ScriptMethodAttribute tells the asmx HttpHandler to output in JSON format.

Assuming (incorrectly, as I later discovered) that I had found a rather silly bug in the Mono class library, I tested the exact same code in Microsoft's ASP.Net...which proceeded to work as expected.

After a bit of a break, I started thinking about how, exactly, MS got the old asmx HttpHandler to output JSON without modifying the existing class library. After all, when the smart folks as MS started adding in the 3.0 and 3.5 class libraries, they vowed to not change the existing class library if it wasn't truly necessary.

I took another look at my web.config file in Visual Studio and discovered what magic made this possible:


<httpHandlers>
    <remove verb="*" path="*.asmx"/>
    <add verb="*" path="*.asmx" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
...
</httpHandlers>

In Visual Studio, the default web.config unwires the old asmx http handler, then wires up a new asmx http handler that lives in System.Web.Extensions.dll. The old handler is still unaware of JSON, the new handler takes care of all of this instead.

The default web.config MonoDevelop created did not contain these modifications the the httpHandler section. Fortunately, everything about the <add> line was the same except for the assembly version (Really? You got the public key token the same but not the assembly version? Why?). Once I modified web.config, I got JSON output from the asmx.

I know not everything in ASP.Net 2.0 is working in Mono, but its really satisfying to see that the team working on it have implemented the parts that are most interesting to everyday web developers. I'm looking forward to seeing what cool bits the next release of Mono will bring.

ASP.Net MVC Partial Rendering Vs. JSON

2009-03-16T10:47:00.006-05:00

I've been doing a little research on ASP.Net MVC, now that it's in RC instead of "Hey, don't trust what we're releasing because we're still changing a lot of stuff"-Preview Release. I was initially a little miffed with the official ASP.Net MVC site, because in iteration #7 of their tutorial, they "Add Ajax functionality" by converting part of their page to an (MVC) ascx control, then partial rendering it when a script request hits it. That's so...ghetto. After all, JSON is the new hotness, right? And what was it that UpdatePanels did that everybody hated? Partial rendering.

That's when a chill went down my spine, as I (as it turns out erroneously) deduced that MS had decided on a One True (ghetto) Way, and the Rails folks would get to continue to mock their .Net brethren for having to jump through huge flaming hoops to just spit out that lean and sexy JSON.

Fortunately for me, I was wrong about everything. Partial rendering in MVC is not ghetto at all. Its actually MVC architecturally correct, light-weight, and templatey-sweet. Also, you can easily return JSON from your MVC controller without so much as striking a match, never mind setting up the gasoline-soaked hoops.

I'm not really interested in retyping the examples in the Asp.Net site, but if you go here and scroll down to "Listing 3". You'll see the tutorial author recycling a controller action to be Ajaxy and not Ajaxy at the same time. Going back up the page a little, you can see what makes their stuff architecturally correct, the Ajax.ActionLink helper, where you specify a route name and don't hard-code your Ajax request URL into script somewhere.

For the vast majority of situations, the tutorial's example will suffice. It keeps you from having to parse JSON on the client, it keeps you from having to loop objects on the client, and as long as you don't get crazy with it and limit the amount of HTML you send back (data paging = good), it should be quite alright on the bandwidth consumption.

As for the JSON, if you need or want it (it is lean and sexy, after all), all you have to do is this:


Public Class SampleController
    Inherits System.Web.Mvc.Controller

    Function Get(ByVal Id As Integer) As ActionResult
        Return Me.Json(SampleDataRepository.Get(Id))
    End Function
End Class

That's it. No having to muck about with WCF or its Configuration Files of Unlimited Confusion +5, no inexplicable PleaseReturnJsonIfYouDontMindSoMuchThanksAttribute(). If you wanted to be strong-typed about it, you could change the function's return type to JsonResult, but if you were going to use the same method to spit out the View and JSON, because you like the word "Get", that may not be very helpful.

I really wish they had covered the JSON alternative in their tutorial. They might have mentioned that hard-coding "/Sample/Get" into your JQuery is naughty in MVC, because it defeats the purpose of routing. They could have also mentioned that you could write in the url from the server side view using Url.Action() from server side code, but they didn't.

Sometimes it makes me happy to be wrong. I like finding out that things will be easier because the MVC team at MS is listening, cares, and did a good job.

Windows Integrated Security in ASP.Net

2009-02-27T10:04:00.006-06:00

I recently saw a story about how the new president of the US was interested in investing millions of dollars in "cybersecurity," whatever all that entails (and being your typical Slashdot reader, I didn't read the actual story). Maybe Mr. Obama's worried about the Borg selling national secrets to the Ayatollah.

We, the app developers, can improve the security of our public facing web apps by spending far less than...wait for it...one million dollars, and the solution is Windows Integrated Security.

Many people think of Windows Integrated Security as the SSO (Single Sign On) solution to save users the trouble of keeping Post-It sticky notes on the underside of their keyboards, and unbelievably inappropriate for web applications that need to be exposed to the public, but Windows Integrated Security is also good for securing Service Processes. You know, like IIS and Sql Server.

If you've never read up about security and performance scaling with web apps, you'll quickly discover in your initial research that using Windows security to authenticate the client, and then impersonating that client to authenticate to the Sql Server is very auditable and secure, but performs poorly under load. I think this is where most people give up and slap a password into their connection string.

Microsoft has a rather dry article on securing distributed applications using the Trusted Subsystem model, but for the typical single-server, single db web app, it boils down to this: use Forms Authentication to validate your user, set >identity impersonate="false"<, and assign your Asp.Net service account to some domain user account that pretty much can't do anything useful on the web or database server. This is easier done than said in newer versions of IIS, since all you have to do is set the user account of the relevant IIS app pool. In 32 bit XP (yeah, 64 bit XP comes with a different version of IIS from 32 bit) and Server 2000, its a bit more convoluted, since the version of IIS that comes with them is Old-N-Busted(TM). You will end up editing that spooky machine.config file to make SERVER\ASPNET stop being Asp.Net's man of the hour.

Oh, and don't forget to run aspnet_regiis -ga [domain account] on your web server, or you'll get mad at it. I really don't know what all, specifically, that command does, except that it "Grants Access" for everything a budding young domain account would need to actually run Asp.Net in IIS.

Once you have your IIS process running ASP.Net under a domain account, its trivial to grant that account whatever access is needed in your Sql Server. And guess what? No passwords in the connection string for some nosy junior coder to be copying down to break your production database. Million dollar cybersecurity, all for free.

EntLib gets serious about IoC

2009-02-24T10:31:00.004-06:00

When I first saw MS's Enterprise Library, I was impressed. EntLib is a high feature enterprise application development framework. It makes ADO.Net suck less. It makes logging suck less. It makes handling exceptions suck less. Now it makes unit testing suck less with Unity.

The Unity Application Block basically does the same thing that Castle Windsor and Spring.Net do, but it appears to be a little more flexible than Spring.Net. One nice thing I noticed is that Unity can create classes even if they aren't specifically configured (unlike Spring.Net), cutting down on some of that janky XML configuration that EntLib seems to have trailing behind it wherever it goes.

In addition to providing a basic IoC container, it provides what the EntLib developers (thankfully) call "Interception" or everyone else (cryptically) calls AOP, or Aspect Oriented Programming. Interception basically gives you the ability to write common code once, point the common code at a set of classes and say things like "When methods on these classes blow up, you run, thereby logging the error."

Of course, as I have mentioned before, using an IoC container is to your benefit, as it encourages you to write code that can be wired up in unit tests or automated integration tests.

After all, the MS EntLib developers recognized the fundamental necessity for these IoC thing-a-ma-jigs:


In general, you should use the Unity Application Block in the following situations:
    * Your objects and classes may have dependencies on other objects or classes.
...

Ummm...who's code DOESN'T have classes that depend on other classes? The COBOL guys?

I haven't experimented with Unity yet to see how well it works with ASP.Net, but apparently David Hayden has, at least with MVC. WebForms will definitely have to be different, you'll probably have to write a special System.Web.UI.Page derived base class to easily gain access to your Unity container (the Spring guys had to for their ASP.Net support). You'll also want to make sure your Unity container lives in application scope, as Mr. Hayden did so creatively.

If you're in a shop that takes advantage of the services of Enterprise Library, you'd be missing a prime opportunity to improve quality and productivity if you're not at least checking out Unity. I mean, its already in the box, you may as well look at it.

NHibernate with PostgreSQL

2009-02-10T22:28:00.010-06:00

PostgreSQL is an open source database server that is sadly ignored in the .Net world. For one thing, its free. For another, it isn't as heavy an install as SQL Server 2005, Developer Edition ($50), which is important to someone like me, who has a laptop with sadly lacking fixed-disk storage capacity. That's back story number one.

NHibernate attracted my attention when the failures of MS Entity Framework left me feeling cheated and the little grievances with Linq To Sql left me wanting for more. Neither are what I would call "nice" when used with a web app, they get pissy about reconnecting an entity fetched on one HTTP request and saved on another, which, interestingly enough, is exactly what every web app known to man does. That's back story number two.

This situation of having two toys I wanted to play with led me to the desire to hook my toys together in a small, contrived solution, just to see if I could figure out how to get them to work.

In my playing, I have discovered a couple of things about both technologies:

PostgreSQL is not SQL Server. They are monstrously different, but the differences can trip you up.

NHibernate is not hard...unless you fail to install the xml schemas into Visual Studio. Without Intellisense enabled on the mapping files, writing the .hbm.xml files quickly goes from trivial to trying.

Open source software deserves its reputation for lacking documentation. That being said, what is there is extremely valuable, and what isn't there is probably in some forum somewhere, if you search for it long enough.

One thing that kept tripping me up over and over again was the way PostgreSQL handles object (schema, table, column, view, etc.) names. For whatever reason, it assumes that all object names are in lower case, even when you type them in mixed or upper case on a query. For example:


SELECT * FROM Books

gets converted to


SELECT * FROM books

I can't even begin to understand why it does this, but it does. Seems like extra work to me, but it is what it is. If you place double quotes around the object names


SELECT * FROM "Books"

then it retains the object names just the way you type them.

Now, that's really not that terrible. You have to place square brackets around object names that SQL Server finds nappy, but I was somewhat offended by NHibernate's PostgreSQL 8.2 "dialect," the plug-in giblets that translate NHibernate's platform-independant queries into the server's native SQL, in that it didn't automatically place double-quotes around everything, whether I needed it or not. Apparently this is a long-standing complaint with this bit of code, I actually found a bug against it that was several years old on the Java version of Hibernate for just this very problem, and yet, there it lies, unfixed. Fortunately, there is another way.

The obvious solution is to add the optional table and column attributes to the mapping files and place backticks (`) around the values. Because .Net developers use backticks every day and that's blatantly obvious to anybody that that is the solution. It has to be obvious to everybody else, because I couldn't find it in the NHiberate documentation anywhere. </sarcasm>

One of the big theoretical advantages to NHibernate is that it abstracts you away from the db vendor by way of its mapping files and query methods (HQL and Criteria Queries). I'm not sure I completely buy into that theory. With what I've experienced, the mapping files would need modifying when moving from SQL Server to PostgreSQL, although none of your VB/C# code would need any attention. Even so, cleaning up all those mapping files would be non-trivial in a large enough existing application.

In conclusion, despite NHibernate's leaky abstractions on PostgreSQL, I could see how a SQL Server developer could come to love using it. It could be a specific boon to those using Asp.Net MVC as well as those who use Web Forms. I look forward to trying it in scenarios where Linq to Sql comes up short and Entity Framework fails miserably.

SQL Server, NULL varchar vs empty varchar

2009-01-28T11:24:00.006-06:00

It always amazes me that people LOVE NULLable fields in database tables. I find very little value in them, as you can represent "empty" as a nullstring (zero-length string) instead, and NULLs tend to make querying the database just a little bit harder to deal with, and what we developers don't really need is for our lives to be any harder.

The only thing that I can figure is that people believe that a NULL varchar in SQL Server takes up less space than an empty NOT NULL varchar. I devised a test to see if this was true.


create table temp(data varchar(500) NULL)
create table temp2(data varchar(500) NOT NULL)

declare @counter int
set @counter = 0
while (@counter < 50000)
BEGIN
 insert into temp(data)
 values(NULL)

 insert into temp2(data)
 values('')

 SET @Counter = @Counter + 1
END
 
exec sp_spaceused 'temp'
exec sp_spaceused 'temp2'

drop table temp
drop table temp2

This script, when run on SQL Server 2005, shows that both tables are exactly the same size, proving that NULLS and empty strings take up the exact same space on the disk.

The moral of the story is this: don't use NULLs if you think its saving you space, its a false economy.

Mono 2.2 drops C# Script on the unsuspecting world

2009-01-21T19:46:00.005-06:00

I discovered an interesting tidbit while perusing the Mono project's version 2.2 Release notes. Apparently, someone on the project hates the inflexibility of C# as much as I do.

Scroll about half way down to the "C# Language" section, and you will discover something... "Compiler Services." Basically, the Mono team has come up with the C# version of the Javascript Eval() function, and an interactive shell, a-la Python or Perl.

This was done to allow applications to provide C# as a macro language. But it could be used to do all manner of interesting things, maybe to make small segments of code that can be modified at runtime, such as business rules processing.

The real question of the day is, can you use it in your MS CLR based projects in addition to using it with Mono based projects?

The nature of Entity Framework is suffering

2009-01-20T20:34:00.010-06:00

In my line of work, the vast majority of my development is web applications to support business operations. Nothing fancy, and rarely anything exciting, but it requires some reasonable code quality and some ability to diagnose and support the running system.

Since these web applications are not usually anything special, they should be relatively easy to create...and yet, writing a DAL for an "enterprisey" web app is tedious as can be imagined. If you follow the conventional wisdom, you have to write sprocs, then write ADO.Net code to run the sprocs, then translate the data output into objects and return them. Then to save, you have to translate objects into sproc calls again.

Just typing the description of all of that makes me tired, so I'm always on the lookout for a way to get the "boss-man" to let me cut out the boiler-plate, carpel-tunnel inducing code typing and get to the job of actually creating quality applications.

My current recommendation, after doing some research, is this:

Please, for the love of everything that is holy, forget stored procedures ever existed. The only thing they add to your application is development cost.

Linq to SQL is a neat curio, but its really harder to use than regular SQL, despite its epic hipness. Use regular SQL instead and save yourself the trouble.

If you're going to use regular SQL, save yourself the annoyance of having to hand-code object mapping in your C#, and generate that junk instead. MyGeneration is free, CodeSmith is cheap, you have no excuse for wasting your employer's money. Just don't use .Net Tiers (of agony).

ADO.Net Entity Framework is probably the worst data access technology I've ever used, sitting at the right hand of Satan. Don't ever use this in a web app. It will only make you angry.

I expect that after I use NHibernate in a production development environment, I will simply replace all of the above recommendation with "just use NHibernate, forget about everything else," but I would have said the same thing about Entity Framework until it punished me for trying to use it.

When I first started playing with EF, I thought it was going to be cool. Then it boxed my ears. In order to use an entity fetched in one web request in another web request, you have to manually detach it from the database context...which isn't Ok. I don't have to do this in Linq to Sql.

Attaching items for update is almost completely impossible with a many-to-many relationship. I never could find the magic incantation to get it to just save an object graph without re-selecting it off the database, then manually updating the object's properties, which sounds suspiciously similar to the whole mapping thing that I don't want to manually do.

Actually, Entity Framework makes many other people angry, but not apparently for the reason it makes me angry.

Entity Framework, its not just for WinForms...oh, wait, yes it is.

IoC you in the morning, Part 2

2009-01-08T09:04:00.004-06:00

In my previous (long winded) post on IoC, I discussed in gory detail how write your own general object factory for obtaining testable objects in a running application.

If you want to know more about why this is even necessary to begin with, read
this Martin Fowler article. I can summarize it shortly for the impatient, however. If your classes are hard-bound together in your application (they create instances of other classes that they need to work directly by using "New") then it makes them nearly impossible to test in NUnit without causing weird (writes bogus data to the db or file system) or performance sapping (reads or writes LOTS of bogus data to the db, then undoes its dirty deeds with a transaction rollback) side-effects (since you forgot to do the rollback) when running unit tests.

Now, if you're like me, and you want to use somebody else's pre-written, pre-tested code to make your app run, the Spring.Net IoC container of Java lineage is probably less annoying than writing your own factory on all but the smallest applications, and in the smallest applications, you probably won't even bother with a factory, you could just set everything up in Application scope or with Shared (static) members somewhere in the UI host code.

Spring.Net's documentation is voluminous to the point of being overwhelming to the beginner, though it is well organized and fairly complete for the more popular sections of the framework.

If you can cull out the portions of the documentation that you don't want, you will find out that Spring.Net's IoC container consists of a class that goes and finds objects and puts them together in a runnable form for you, and a configuration file/section that tells the class how to create those classes and glue them together at runtime. The configuration file is highly flexible, you can place it directly inline in your app.config/web.config, you can place it in an easily accessible external file and access it by Asp.Net url (~/Config/SpringConfig.xml for example) or file path, or you can have it embedded in an assembly and have Spring.Net load its configuration directly from the dll.

You furthermore have complete control over the instancing behavior of the objects. You can specify any given object be created once and retained for eternity, returning the same object instance on every GetObject() call to Spring or created anew per GetObject() request. With web apps, you can further specify that a class is created once per web request, once per web session, or once per web application, with the caveat that when using Session state, your Session state should be set to InProc, or you suffer object serialization (which SUCKS! for performance). The default for the settings is for an object to exist eternally and live in the web Application state. You can also have the container create objects eagerly, without waiting for you to ask for them, or "lazy-init" them, where it doesn't create them until it absolutely needs to do so. The default is to create objects in advance.

I usually have the container lazy-init objects and retain them for the lifetime of the web request only for my web apps. Its just easier to write code against this instancing model, and it is relatively easy on the hardware's resources. This isn't the MAXIMUM SCALABILITY solution, but when that becomes a requirement, you'll be refactoring code anyway.

For people who develop in Asp.Net, there are additional goodies in Spring.Net to make life easier. There is a base page provided by Spring.Net that derives from System.Web.UI.Page, unsurprisingly called Spring.Web.UI.Page that provides easy access to the container class. There is some additional configuration that must be done to make Spring.Net set up its base page correctly, but with a little copy-and-paste from their documentation, setting up the Asp.Net extensions are a snap, and the Spring.Net base page includes its own ghetto-MVC data binding that you can use or not as you see fit.

I know this post is pretty shy on code, but Mr. Fowler seems to have already done an excellent job of explaining the details of using Spring. I hope Spring.Net makes your software suck less, as it has with mine.

A legitimate use for temporary tables

2008-12-03T13:32:00.007-06:00

Temp tables are those monstrosities that start with # in SQL Server. They exist in a netherworld in between your database and the TempDB database, and they only exist for the current connection...unless you're crazy enough to use a ## temp table, then they're in a global TempDB netherworld...then they mysteriously break your code.

In general, these things are rightfully maligned because they usually just cause way more trouble than they're worth, especially when an error occurs in the database that somehow ends up with the temp table not getting DROPped. Conventional wisdom says to use table-valued variables instead...and yet, if you're trying to load half-a-million rows into a table variable, you're hosed.

Here's the scenario where the evil little buggers actually work: bulk loading a flat file into a relational structure through TSQL because SSIS is (still and always will be) crap:

Create an XML Format File to describe your flat file to the DB server

Create a temp table wiht columns that you like from your file

INSERT INTO #EvilTempMonster ... FROM OPENROWSET(BULK 'file name goes here' FORMATFILE='format file name goes here')

Place indexes on your temp table

Load the relational structures to your heart's content.

Drop the temp table

Do your bulk load outside of a transaction (you aren't going to modify your file from inside SQL), but do all of your relational INSERT/UPDATE stuff in a transaction, wrapped in a TRY...CATCH block, and on catch, ROLLBACK.

It should look something like this:


DECLARE @ErrorMessage nvarchar(MAX)

--Just make sure your temp table can't somehow
-- screw with you and ruin your day.
IF object_id('tempdb..#MyFlatFile') IS NULL
    DROP TABLE #MyFlatFile

CREATE TABLE #MyFlatFile(...columns rock...)

--Mmmm, much better than BULK INSERT.
--Even if you have to make this dynamic SQL to
-- vary the file name and format file name.
INSERT INTO #MyFlatFile(...columns again...)
SELECT ...flat file column list...
FROM OPENROWSET(BULK 'filename.txt' FORMATFILE='formatfile.xml')

CREATE CLUSTERED INDEX IX_ImportantMatchField
ON #MyFlatFile (ImportantId)

BEGIN TRY
  BEGIN TRANSACTION

  INSERT INTO TableA
  FROM #MyFlatFile
  WHERE #MyFlatFile.ImportantId NOT IN (SELECT DISTINCT MyImportantId FROM TableA)

  ...Other inserts/updates...

  COMMIT
END TRY
BEGIN CATCH
  ROLLBACK
  SET @ErrorMessage = ERROR_MESSAGE()
END CATCH

DROP TABLE #MyFlatFile

IF @ErrorMessage IS NOT NULL
  RAISERROR(@ErrorMessage, 16, 1)

At least, that's one way to do it. You could just do the DDL and bulk load inside the transaction too, and that would be easier on the programmer, I'm just unsure of the performance implications.

With my script, I can load half-a-million rows into 3 different related tables, reducing out duplication, scrubbing out double-quotes (thanks Excel), inserting new rows and updating existing rows in under 3 minutes on what is a pretty "slow" development server.

Why go through all of this when SSIS has a pretty graphical designer? My IS packages would not execute in the context of my web server, running under a least-privileged-user account (whoops!). Furthermore, I could not use transactions within the IS packages because of permissions problems involving the Distributed Transaction Coordinator. WTF is it using that for? SSIS COM at the core. Awww, yeah! That's what I'm talkin' about!

Has anybody else found a use for temp tables that made you move them from the "Evil-Never Use" category to the "Correct tool for some circumstances" category?

How to make delimited strings in C#

2008-11-12T09:34:00.006-06:00

In prior popular MS programming languages and within the COM runtime environment, strings were mutable that is, the program could change the contents and size of a string in memory without having to copy the whole thing over in memory.

In the world of .Net, though, strings are immutable, which means once they are made, any changes to them causes them to be completely copied in memory to be modified. As you can imagine, this is quite painful, performance-wise. This is why the .Net runtime has a StringBuilder class, which mimics the effects of real mutable strings.

Now, in mutable string languages, building a delimited string quickly was as simple as doing this:


Dim delimiter = ""
Dim delimitedString = ""

For Each Item In intArray
    delimitedString = delimitedString & delimiter & CStr(intArray)

    'Single char assignments are super cheap
    ' because the string has already been allocated.
    delimiter = ","
End For

Do that in VB.Net or C#, and it'll run like poo, because every time the delimiter is assigned, a new string is allocated in memory and then its value set, versus what VB6 did, which was to just change char[0] of the delimiter string to a comma...over and over and over again.

Here's how you do the same thing in a .Net language and retain performance:


if(intArray == null || intArray.Length == 0)
    return string.Empty;
else
{
    StringBuilder sb = new StringBuilder(intArray[0]);

    for(int itemCounter = 0; itemCounter < intArray.Length; itemCounter++)
    {
        sb.AppendFormat(", {0}", intArray);
    }

    return sb.ToString();
}

In VB:


Return Join(intArray, ", ")

I couldn't tell you what kind of performance the VB Join() method gets (and I have no doubt writing a loop yourself is faster), but you can't argue with the ease of it. And before you say C# can use string.Join(), you have to know that string.Join() requires its input array to already be an array of strings, which would have required me to loop through the int array above and make a string array just to call string.Join(). Which would be stupid. Because you never loop over a data set twice when once will do the trick.

Linq to Magic Smoke

2008-11-03T18:31:00.005-06:00

The sky is falling! The sky is falling! Cats and dogs, living in sin!

Microsoft let the Magic Smoke out of Linq to SQL.

I don't really understand all the noise about it. All they said is that its not going away, but its not going to get better, either, so if you wouldn't so much mind, please use ADO.Net EF instead.

I saw some blog posters (the most vocal and afraid) who were bemoaning the apocryphal demise of doing that easy, easy data access that Linq to Sql has impressed me with.

Now take a quick peek at what the MSDN article says of Linq to Entities in conjunction with ADO.Net EF
here. Drink it in.

If you didn't behold the coolness therein, I'll summarize. You can do everything you could do in Linq to Sql, and it pretty much works in code the exact same way. You probably won't even have to change that much of your code to switch it over.

Here's what you get in addition:

Entity SQL
- Like HQL from NHibernate, this SQL-ish language gives you the flexibility to do just about anything you need with the database server, including server side paging and sorting. Its vendor independent, as well, which should really rile the guys at Oracle. T-SQL to query Oracle? Preposterous! If you hate it, don't use it. Nobody will hate you if you don't.

Redefine entities that don't make any sense
- Some legacy data structures are insane. Now you don't have to deal with that garbageoutside the DAL! Write Business logic components...*NOW! with less suckage!* for your poor web developers to consume.

Join to that DataSet
- Many legacy DALs...ok, many current DALs...use ADO.Net DataSets. Datasets are great, so a goodly number of DALs used them. Now you can write new Business Logic, using both new and old data sources, because rewriting entire working DALs tends to cost heaping sums of cash, but you don't want to use the old-n-busted of the DataSet world.

So everybody calm down about Linq to Sql. MS doesn't want to waste money on developing two technologies that basically do the same thing, and they're not taking the keys to the Camero. Of course, if they really wanted to save money, they would have written a hook to Linq for NHibernate.

Stay tuned for MS "inventing" an IoC container instead of telling people that Spring.Net is for lusers.

IoC you in the morning, Part 1

2008-10-31T11:31:00.014-05:00

There's been quite a bit of harping lately on how everybody should, at the very least, be writing testable code, if not following the principles of Test Driven Design. The fact of the matter is, you SHOULD be writing testable code, but nobody has ever make a concerted effort to put all the pieces together in a comprehensible without using make up words and phrases that make easy things hard to understand.

The first thing you want to do when you are writing testable code, is to figure out which classes you are interested in testing. Enterprisey code (if it isn't crap) is layered, like a cake. On the bottom is your Data Access Layer (DAL), that keeps all that funky SQL crap out of the next layer up. The next layer, then, is your Business Logic Layer. It does the work. The icing on the top is the UI Layer, which in the case of this posting will be ASP.Net. Not that there's anything wrong with that.

Inside the different respective layers, you can conceivably have more slices and layers if it helps achieve your ends...though most people don't. Now, if your Enterprisey web app is laid out as above, then most of your actual work (mucking with files, interacting with external subsystems, manipulating data...the fun stuff) should be in your middle layer, your Business Logic Layer. This is the layer you want to test.

Now, sure, you can grab a copy of NUnit (my unit testing poison of choice) and go to town, but where do you start? First of all, all of your BLL classes need to live in assembly(ies, whatever) separate from your UI. Web projects/web sites are completely untestable from the NUnit perspective, because NUnit needs a dll to bind to. Just make your BL a new project in your solution, set a project reference to it, and you'll never know the difference.

The next step is to make your BL classes less aware of classes that they depend on. This means that you have to stop creating DAL objects within your BL classes, and provide some mechanism to attach those DAL objects you need to the BL classes.

There are a few ways to make that attachment. One is to just make a constructor that takes all the DAL components. That's for some things, but if you have a lot of DAL objects affixed to a single BL component, it gets pretty harry.

VB:


Public Interface IDoPeople
 Function GetMyPeeps() As PersonEntity()
End Interface

Public Class PersonDataAccessObject
 Implements IDoPeople

 Public Function GetMyPeeps() As PersonEntity() Implements IDoPeople.GetMyPeeps
   'Do Data Access Here
 End Function
End Class

Public Class PersonManager
 Private MyDAO As IDoPeople

 Public Sub New(ByVal DAO As IDoPeople)
     MyDAO = DAO
 End Sub
End Class

C#:


public interface IDoPeople
{
 PersonEntity[] GetMyPeeps();
}

public class PersonDataAccessObject : IDoPeeps
{
 public PersonEntity[] GetMyPeeps()
 {
    //Do data access stuff here.
 }
}

public class PersonManager
{
 IDoPeople MyDAO;

 public PersonManager(IDoPeople DAO)
 {
    MyDAO = DAO;
 }
}

Now, imagine that, with 8 different DAO arguments on the constructor. It works, but its ugly. I much prefer to use properties for DAOs and other dependancies, like this:

VB:

Public Class PersonManager
 Private _MyDAO As IDoPeople

 Public Property MyDAO As IDoPeople
     Get
         Return _MyDAO
     End Get
     Set(Value As IDoPeople)
         _MyDAO = Value
     End Set
 End Property
End Class

C#:

public class PersonManager
{
 public IDoPeople MyDAO {get; set;}
}

That's just my preference. I actually have a technical reason for preferring properties, but for the most part, it doesn't matter which way you do it.

Also note the interface. It is the key to making your Business Logic class testable. The interface is basically a definition of what the BL class will accept as a Data Access Object. It doesn't necessarily have to be an instance of PersonManagerDataAccessObject, it could be anything that just kinda looks like the PersonManagerDataAccessObject. That means that the MyDAO property above is mockable.

The next issue to address isn't really how to test your code, but how to actually make the code work in the real system at runtime. I mean, your BL component's properties have to get assigned, or the BL component won't work, right?

I wrote a static class with a single static method on it to go through creating Business Logic components for me.
VB:


Public Shared Class ObjectLocator
 Private Shared InjectableTypes As Dictionary(Of Type, Type)

 Public Shared Sub New()
     InjectibleTypes = New Dictionary(Of Type, Type)()

     InjectibleTypes.Add(GetType(VampireBasic.Data.IDoPeople, GetType(VampireBasic.Data.PersonDataAccessObject))
 End Sub

 Public Function GetManagedObject(ByVal objectName As String)
     'Create components here, or fetch them from the cache.
 End Function
End Class

C#:

public static class ObjectLocator
{
 private static Dictionary<Type, Type> InjectibleTypes;

 public static ObjectLocator()
 {
     InjectibleTypes = new Dictionary<Type, Type>();

     InjectibleTypes.Add(typeof(VampireBasic.Data.IDoPeople), typeof(VampireBasic.Data.PersonDataAccessObject));
 }

 public static object GetManagedObject(string objectName)
 {
     //create components here, or fetch them from cache.
 }
}

The contents of GetManagedObject() are pretty complicated, but it basically does the following:

Go see if the object is in the per-http request cache (look at HttpRequest.Current.Items)

If its not in the cache, go see if its really something we can try to create...that means reflection. Its ok to use reflection for this, because the resulting object will be cached for the rest of the request. All reflection costs are paid only once
per object lookup per http request.

Once its created, pull all of the properties off that are an interface of some sort--that's the technical reason why I used properties, because I know how reflection works for properties, and its pretty easy to deal with.

Go through the list of interfaces, and try to pull them from HttpContext.Items cache. If its not there, check with the InjectableTypes dictionary. If the interface is in there, create an instance of the type it is associated with and cache it.

Now, this is the part where it gets a little specific to my task. I'm using LINQ, so there is a property on every InjectableTypes DAL object called "Database" that is derived from LINQ's DataContext class. Since I know what that is, I just create ONE AND ONLY ONE DataContext and attach it to the DAL object. Then cache the DataContext in HttpContext.Items.

The whole point of the ObjectLocator in my project is to cache the DataContext object. There are two reasons for this. If you query LINQ for an entity more than once on the same DataContext object, LINQ will not query the database again for that entity, improving performance. Second reason is transactions, which pretty much require you to keep your database connection open during the scope of the transaction. If you do something like this:


public class TransactingBLComponent
{
 public ILikeMonkeys MonkeyDAO;

 public void DoSomethingComplex
 {
     Using(System.Transactions.TransactionScope txn = new TransactionScope())
     {
         MonkeyDAO.DoOneThing();
         MonkeyDAO.DoAnotherThing();

         txn.Complete();
     }
 }
}

and you don't keep your database connection open for the duration of the TransactionScope, all manner of annoying things occur (MSDTC elevations, yuck).

Since I don't create a new DataContext with every call to the MonkeyDAO, transactions work as they should.

So, that's how I manage dependencies between objects in an Enterprisey web app. Of course, you could just use Spring.Net, but where's the fun in that?

Formal Properties

2008-10-15T09:19:00.004-05:00

Ahhh, yes. Formal Properties in a programming language are something that all mainstream .Net (and Delphi) developers have come to know and love. It is quite easy to take something so simple for granted, especially if you are used to programming in .Net languages.

Its also very easy to miss them when you switch to other languages that do (or did) not support formal properties. Take Objective-C, for instance. It has been knighted by Apple as the "core" language of the Mac OS platform, and has been fairly well regarded by folks who can get past its lack of Namespace support and its rather odd method call syntax. It pretty fast and if Obj-C doesn't have the features you need, you can simply include a regular, run-of-the-mill C or C++ header file, and continue on your way. What it didn't have before Objective-C Version 2.0 was Formal Properties.

You can't possibly believe the ruckus adding formal properties caused in the Objective-C community. It would have been similar to putting a directive in C# called #SCREW_STRICT_TYPING (which I personally thing should happen) to turn on dynamic language-style late binding. The whole thing is, now you can get the Obj-C preprocessor to synthesize properties for you, much like with C#'s oh-so-sweet automatic properties. That isn't what ticked off the purists, though. They changed the syntax for accessing properties. It used to work like this:


MyWonderfulClass *wonderfulClass = [[[MyWonderfulClass alloc] init] autorelease];

//Set a property here.
[wonderfulClass setCoolProperty: @"Valuable Data"];
//Get a property value here.
NSString *data = [wonderfulClass coolProperty];

And that still works great, if you know the pattern. If you created getters/setters and didn't follow the pattern, the development tools wouldn't see your properties. The new syntax in Obj-C looks just like it does in C# or VB. You access the properties like this:


MyWonderfulClass *wonderfulClass = [[[MyWonderfultClass alloc] init] autorelease];

//Set a property here.
wonderfulClass.CoolProperty = @"Valuable Data";
//Get a property here.
NSString *data = wonderfulClass.CoolProperty;

And that is what caused all the fuss. Some folks though it was too...C#-ish, others just thought it was ugly, and out of character for the Objective-C language. Some thought it was great, because they didn't have to type square brackets for properties anymore.

Now, for those of you who are more C#-ish and don't really understand all of the Objective-C square-bracket mania, I'll give you the short C# low-down. The first line in the code samples is similar to saying this in C#:


using(MyWonderfulClass wonderfulClass = new MyWonderfulClass())
{

}

Specifically, the square brackets indicate the Objective-C equivalent of a function call in C#, except Objective-C doesn't do direct function calls on its own objects. It will for ANSI C functions and C++ functions, but Objective-C objects are special. Objective-C sends a message to the object and the object responds. I thought that was all conceptual mumbo-jumbo when I was first learning it, but as it turns out, it really does send messages to the objects and the objects respond. If you want to know more about how Objective-C works, see this article on Wikipedia, and if that really isn't enough for you, see theauthoritative source at Apple's developer network site.

Now, back to properties. In the Java world, they do what they did in old-school Objective-C (and C++ before that). Specifically, for properties you should follow a pattern for your getter/setter functions. In Java parlance, it is called Bean pattern. And just like everything Java, the documentation makes it out to be a whole lot more complicated than it really is. It works by defining the expected signatures for "property" getter and "property" setter functions so tools like IDE's can figure out what the heck you're doing. The problem with that approach, as far as I can see, is that if you don't know about the pattern, you are going to be frustrated with it until you either figure out what that pattern is, or see Eclipse or NetBeans do it for you, and if you see the IDE do it for you, you may not understand why it does things that way, which I consider to be a distinct problem.

Contrast to VB, C#, and now Objective-C, where you have the pattern hard-wired into the language. You know what a property is when you glance at it (in VB because it has the Captain Obvious keyword of PROPERTY plastered all over it). Formal property support is even engrained into the CLR itself. All you have to do is look at how you pull properties out of an unknown object to see how very formalized properties are in the .Net world.

Basically, the philosophical difference between formal and informal properties is the same philosophical debate that Ruby on Rails and subsequently ASP.Net MVC brings to the world of web development. Do you like Convention over Configuration? Do you prefer formalized, compiler checkable specifications?

Personally, I prefer formal properties the way they are implemented in .Net over what Java uses, because I can save myself some icky-looking parens when I chain together property-gets and method on the ends of properties, but that is, admittedly trivial and wouldn't stop me from using Java if I thought it had a clear advantage over .Net for development. The best way to decide, is of course, to use the different languages and make the choice for yourself.

Have fun coding!

Language of Choice

2008-09-30T09:52:00.010-05:00

Visual Basic used to be the red-headed stepchild of the Microsoft world. It was the language we all loved to hate. It had enough bondage and discipline (you can't put parenthesis on a Sub invocation unless you put the Call keyword in front of it...WTF?) in the days of VB6 and lacked enough performance and advanced features that it basically wore a sandwich board with a target painted on it.

Then along came the CLR, and the .Net label was affixed to VB. Gone were the B&D features (Get/Set/Let for properties? WTF?). Gone was the mediocre performance. Gone was the love. Even though VB 7 became what Borland wished Delphi could have been, an easy-to-read language that eschewed some of the more...dangerous features of the C-derived languages whilst giving the developer enough room to be creative, and enough performance to avoid new hardware purchases, many developers think the language is total, utter crap.

The sad fact of the matter is, LoC is many times an emotional decision. Developers love to hate VB (or Java for that matter) because of its un-hipness. They hate being mocked because language X doesn't have wicked cool feature Y.

For example:

Dev A: You use What? It has no clojures. It sucks.
Dev B: WTF? I don't know what a clojure is, and I probably don't need it.
Dev A: [Emits toxic levels of Smug]

This doesn't excuse you from knowing which languages are capable of what features. You never know what feature you could use to save your company cash and make you look like a wizard.

For example, see C# run straight into a brick wall with this bit of love:

...

object listOfCritters;
if(gimmieHTML)
{
listOfCritters = GetArrayOfLions();
}
else
{
listOfCritters = GetArrayOfMonkies();
}

foreach(var animal in listOfCritters)
{
if(animal.TypeName = "Monkey")
animal.FlingPoo();
else
animal.PreyUponGazelle();
}

...

Obviously, this won't work in C#. The compiler will verbally abuse you for trying to make some undefined, random call on some untyped, random, unknown object. This never really made much sense to me, because the runtime engine will know what type it is, and if you write good unit tests, you'll know well before deployment if you're code is bonkers or not, but it is what it is.

Next, in VB:

...
Dim listOfCritters
If gimmieHTML Then
listOfCritters = GetArrayOfLions()
Else
listOfCritters = GetArrayOfMonkies()
End If

For Each animal in listOfCritters
If animal.TypeName = "Monkey"
animal.FlingPoo()
Else
animal.PreyUponGazelle()
Next

...

This is contrived, of course. For a realistic example, in SMO, the authors attached the Name property to table objects, stored procedure objects, view objects, etc, at the final object level. If you get a list of "Database objects"(I don't remember what their base class really is), with tables and sproc's and what-have-you all mixed in together, there is no interface to fall back to to make accessing the Name property generic.

Point being, know which languages have which features. Don't be afraid to learn all 3 of the .Net biggies (VB, C#, C++/CLI). With the economy getting soft and flabby around the middle, it may make the difference between you paying the rent and paying the unemployment office a visit.

Sub Main()

2008-09-22T08:31:00.003-05:00

Vampire Basic - this name is the result of a joke about a throw-away utility I once wrote in VB.Net on a project that was entirely C#. For whatever reason, I had come in early and had left the lights off in the office, so when the next person came in and saw it was VB, I told them that I was coding in the dark so it wouldn't die from exposure to the light, like a vampire.

In this blog, I will address thoughts and issues covering multiple platforms and multiple languages including C, Obj-C, Java, C#, and of course, Vampire Basic.Net. The first post with real, usable content should appear about one week from now. I haven't decided exactly where I am going to start, but one of my goals is to promote expanding skill sets across language and even platform boundaries.