Modern Signal Development Blog

I wanted to write about another bit of code that I created for the What Color Is...? website I talked about in my previous post. 

Displaying the Results

Probably the most interesting and challenging part of this project was to come up with a way to generate and display the results in an efficient way.  Colors were all saved in the database as 3 separate fields (Red, Green, and Blue).  This made it easy to select a color average for all responses.  Here is the SQL:

SELECT  t.name ,
        COUNT(*) AS Count,
        AVG(IsNull(red,255)) as Red ,
        AVG(IsNull(green,255)) as Green ,
        AVG(IsNull(blue,255)) as Blue
FROM    WCIResponse r
        INNER JOIN WCIThing t ON r.ThingID = t.ThingID
GROUP BY t.name

One of the options in the survey is to select "No Color", which is saved in the database as null values.  These are converted to 255 so that it is the same as if they selected white.

That was pretty easy, but I also collect some basic, anonymous demographic information to enable breaking down the results a bit.  I ask for age, for instance.  What I wanted to do was to break down the results by age group, i.e. 20-29, 30-39, etc.  This is more complicated, of course, and I didn't want to get into doing a massive number of queries and loops, etc, to build up the results.  Fortunately I have recently gotten to know more about the SQL Server statement PIVOT.  PIVOT is made for creating just such reports as this.  This is somewhat more complicated than a simple PIVOT, because I don't want a column for every age, but I want to group the ages by decade.

I'll run through the ColdFusion/SQL code I used to make this work. 

First, I set a variable with an SQL expression to create the name of the age group, e.g. "40-49":

<cfset var AgeRange = "CONVERT(VARCHAR,Age/10*10) + '-' + CONVERT(VARCHAR,Age/10*10+9)">

I only want to include the age ranges for which I actually have data, so I do an initial query to get the age ranges in the data.  Note that the AgeRange is wrapped in brackets so that it can be used as the column name in the next query, e.g. "[40-49]".

<cfquery name="GetAges" datasource="#application.dsn#">
    SELECT  DISTINCT
            '[' + #preserveSingleQuotes(AgeRange)# + ']' AS AgeRange
    FROM    WCIPerson
    WHERE   Age IS NOT NULL
    ORDER BY AgeRange
</cfquery>

I can then build the PIVOT query including all of the age ranges.  I'm still not comfortable with the PIVOT syntax.  I've only used it a few times, and it's still hard for me to wrap my head around without looking at documentation and examples.  If you're new to PIVOT too, this site has a few good examples.  As you can see below, I reuse the AgeRange variable I set above twice.

<cfquery name="GetResults" datasource="#application.dsn#">
    SELECT  t2.name
            <cfloop query="GetAges">
                ,ISNULL(t2.#GetAges.AgeRange#,'0,255,255,255') AS #GetAges.AgeRange#
            </cfloop>
    FROM    ( SELECT    t.ThingID ,
                        t.name ,
                        #preserveSingleQuotes(AgeRange)# AS AgeRange,
                        CONVERT(VARCHAR, COUNT(*)) + ','
                        + CONVERT(VARCHAR, AVG(ISNULL(red, 255))) + ','
                        + CONVERT(VARCHAR, AVG(ISNULL(green, 255))) + ','
                        + CONVERT(VARCHAR, AVG(ISNULL(blue, 255))) AS Color
              FROM      WCIResponse r
                        INNER JOIN WCIThing t ON r.ThingID = t.ThingID
                        INNER JOIN WCIPerson p ON r.PersonID = p.PersonID
              WHERE     p.Age IS NOT NULL
              GROUP BY  t.ThingID ,
                        t.name ,
                        #preserveSingleQuotes(AgeRange)#
            ) AS t PIVOT ( MAX(color) FOR AgeRange IN ( #ValueList(GetAges.AgeRange)# ) ) AS t2
</cfquery>

Although the What Color Is...? site doesn't have a lot of data at this point, I believe this code should scale pretty well.  Of course in the unlikely event that the site becomes an internet phenomenon and gets millions of responses I may have to revisit this code.

Comments (0)   Posted on January 31, 2011 5:09:23 PM EST by David Hammond

I recently created a fun little weekend project (well, an MLK Day project to be more specific).  It's a simple survey to query users about what colors come to mind when they think about abstract ideas.  It's called What Color Is...?  The original motivation behind the site was to come up with an idea for my daughter's science fair project.  She sees letters and numbers as colors (something called grapheme-color synesthesia).  I came up with the idea to devise a simple test to see what colors, if any, most people see when they think about an abstract idea.  From there I naturally began thinking about how it could be turned into a slick online survey.  It was an interesting little project that had a number of minor technical challenges along the way.

A Touch of Eye-candy: Colorify jQuery Plugin

For the design of the site, I chose a basic gray jQuery UI theme (Smoothness).  I did this so as to not distract from the main point of the survey by putting too much color into the surrounding UI.  I did feel like adding a little bit of fun color into the UI though.  I decided that when I would put in some multicolored text in the page titles.  To achieve this, I ended up creating a simple jQuery plugin that takes any element and assigns a string of random colors to the letters in the text.  I called it "colorify":

(function($){
    $.fn.colorify = function(){
        var cify = function(s){
            var rgb = function(){
                var i = Math.floor(Math.random() * 3);
                var r = function(n){
                    return n == i ? 0 : Math.floor(Math.random() * 256);
                }
                return "rgb(" + r(0) + "," + r(1) + "," + r(2) + ")";
            }
            var s2 = "";
            for (var i = 0; i < s.length; i++) {
                s2 += "<span style=\"color:" + rgb() + ";\">" + s.substr(i, 1) + "</span>";
            }
            return s2;
        }
        this.html(cify(this.html()));
    };
})(jQuery);

This goes through each letter in the text, gets a random RGB value and assigns it to the letter by wrapping a span tag with a style parameter around the letter.  It's used like any simple jQuery plugin:

$("#wcititle").colorify();

Which creates colored text like this:

What Color Is...?

One interesting thing about this is that my first attempt at the function resulted in some colors that were just too light to show up well on a white background.  That's an obvious problem when getting completely random colors.  The solution I found was to randomly choose one of the 3 color components (R, G, or B) and set that to 0 (full saturation).  This preserves the generation of bright colors without making them too light.

I wouldn't call this a very useful plugin -- I'll probably never use it for another project.  But it was kind of fun to write.  Check it out in action at What Color Is...?

Comments (0)   Posted on January 31, 2011 4:44:37 PM EST by David Hammond

I just implemented something that should have been simple, and actually is simple, but which ended up being a bit more of a struggle than it should have been.

We have a site where occasionally we were getting duplicate payment transactions when people submitted orders.  The transactions were all within a couple seconds of each other, so it appeared that people were just double-clicking the submit button (despite, of course, a message on the page saying to only click the button once).  We did, once upon a time, have javascript code in place that disabled the submit button when it was clicked once, but that code has been removed somewhere along the line.  I think it had been disabled due to the difficulty of dealing with client side validation -- i.e. if the client-side validation failed, the button should not be disabled, because then the user will not be able to submit the form.

I found a variety of posts online about dealing with this problem.  The basic solution is simple, which is to insert code into the form submit handler that runs the client-side validation and only disables the button if it passes.  First I set up this javascript function:

<script type="text/javascript">
    function handleSubmit() {
        if (typeof (ValidatorOnSubmit) == 'function' && ValidatorOnSubmit() == false) {
            return false;
        } else {
            $("form#aspnetForm input[type=submit]")
                .click(function() { return false })
                .fadeTo(200, 0.5);
            return true;
        }
    }
</script>

Then in the page's code behind, I registered the onsubmit function like so:

Page.ClientScript.RegisterOnSubmitStatement(Me.GetType, "OnSubmitScript", "return handleSubmit()")

The javascript function runs the same code that ASP.Net automatically runs for client-side validation, and, if the validation passes, uses jQuery to disable the submit button and fade its appearance.  The fading isn't necessary, but since it's so easy to do with jQuery, why not?

There is one thing in there though, that threw me for a loop.  Most of the posts I found online set the disabled parameter for the button, which you could do with jQuery like this:

$("form#aspnetForm input[type=submit]").attr("disabled", "disabled");

But when I did that I found that the submit action tied to the button no longer fired on the server side.  After puzzling over this for a while, I realized that when the button is disabled, the button value is no longer sent along with the form parameters, and thus the server-size click event for the button is not fired.  I wasn't able to find any references to this problem online (hence the impetus for this blog post).  So instead of using the disabled parameter, I set the click event of the button to return false, which seems to work well.

Something else that I like about this solution, is that it is very generic.  Since I use a generic selector to get the submit button ("form#aspnetForm input[type=submit]"), the javascript function could be included in a site-wide javascript library, and the onsubmit function could be registered in a master page to enable this functionality for all submit buttons on a site.  I'm not ready to do that on this site, since I just want to make sure the order checkout works well for now, but it's nice to know that if this comes up again, I can fix it in a flash.

Comments (0)   Posted on November 5, 2010 3:22:46 PM EDT by David Hammond

I wanted to take a few minutes to talk up a cool little tool that I started using a few weeks ago and have since come to rely upon.

smtp4dev (http://smtp4dev.codeplex.com/) is a dummy smtp server that, instead of sending email messages, stores them in a list for easy viewing and inspection.  Before I started using this tool, any time I wanted to test email sending for an application I had to, first of all, sanitize all of my data to make sure I didn't accidentally send messages to people that shouldn't get them, and then make sure any messages I needed to inspect were sent to my email account.  I would have to wait for messages to arrive in my inbox and deal with all of the test messages getting mixed up with my regular mail.  It was a big hassle, in other words.

An added benefit for me is that, since I access my regular mail through Gmail, I am able to easily open up the messages in Windows Live Mail (also a free download) which seems to replicate the way the email displays in Outlook.  For highly styled messages this is very important, since Outlook is a very popular client that has quite a few display quirks.

Comments (0)   Posted on July 29, 2010 9:45:06 AM EDT by David Hammond

We had a client trying to upload a 44mb file, and it was failing.  The ColdFusion page that that did the upload had a very high timeout setting, so I didn't think that timeout was the issue.  The ColdFusion administrator had a request limit set of 100MB (which I believe is the default).  What was up? 

It took a bit of googling to realize that IIS7 (the webserver on our Windows 2008 Server), had a default request limit of 30MB.  The solution is rather simple.  With IIS7, a web.config file can be used to set a variety of webserver settings, so I added one to the site that was having the problem.  Here are the contents of the file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <security>
            <requestFiltering>
                <requestLimits maxAllowedContentLength="1024000000"/>
            </requestFiltering>
        </security>
    </system.webServer>
</configuration>

This sets the request size limit to 100MB, to bring it in line with ColdFusion.  This isn't the only thing that can be controlled with web.config.  For some information on how to tighten security on a site using web.config see this page: http://www.petefreitag.com/item/741.cfm

Comments (0)   Posted on May 3, 2010 1:44:47 PM EDT by David Hammond

Some background first...

I had the need to do diff of html content in a project I was working on, which brought me pretty quickly to DaisyDiff, a really nice Java-based utility.  DaisyDiff doesn't however, have a simple built-in function to do a diff of two strings.  There is a command-line option, which takes the paths of two files as arguments, and also a java api that take a number of java objects as arguments.  What I wanted was a function that took two strings and output the results, but DaisyDiff has no such simple function.

I don't really do java development -- that is I've done some in the past but it's been a while and it would probably take me some time to get my development environment up to snuff.  Besides, I didn't really feel like dealing with compiled code. 

A quick google search, of course, turns up CFX_CompareHTML and the JavaLoader version of the same thing.  So I used that, and it worked fine.  But it was using an old version of DaisyDiff, and it seemed to have some bugs with UTF characters and such.  What I really wanted to do was to use JavaLoader to load the current version of DaisyDiff.  After much stumbling around in the code, I found that the test suite in the DaisyDiff repository has exactly the function I wanted -- it compares two strings and returns the result.

So, long story short, I took the code from that function and pulled it into a CFC, using JavaLoader, and rewrote everything in CFML.  The result is the simple function I was after.

So anyway, here it is:

<cfcomponent hint="Wrapper for DaisyDiff" output="false">

    <cffunction name="Init" output="false" returntype="DaisyDiff">
        <cfargument name="daisydiffpath" hint="absolute path to daisydiff jar file" type="string" required="true">
        <cfargument name="javaloaderpath" hint="component path to JavaLoader.cfc" type="string" required="true">
        <cfset This.daisydiffpath = arguments.daisydiffpath>
        <cfset This.javaloaderpath = arguments.javaloaderpath>
        <cfreturn This>
    </cffunction>

    <cffunction name="Diff" output="false" returntype="string">
        <cfargument name="olderHtml" type="string" required="true">
        <cfargument name="newerHtml" type="string" required="true">

        <cfset var paths = [This.daisydiffpath]>
        <cfset var loader = createObject("component", This.javaloaderpath).init(paths)>
        <cfset var TransformerFactoryImpl =     loader.create("com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl")>
        <cfset var StringReader =                 loader.create("java.io.StringReader")>
        <cfset var StringWriter =                 loader.create("java.io.StringWriter")>
        <cfset var Locale =                     loader.create("java.util.Locale")>
        <cfset var StreamResult =                 loader.create("javax.xml.transform.stream.StreamResult")>
        <cfset var OutputKeys =                 loader.create("javax.xml.transform.OutputKeys")>
        <cfset var NekoHtmlParser =             loader.create("org.outerj.daisy.diff.helper.NekoHtmlParser")>
        <cfset var DomTreeBuilder =             loader.create("org.outerj.daisy.diff.html.dom.DomTreeBuilder")>
        <cfset var HTMLDiffer =                 loader.create("org.outerj.daisy.diff.html.HTMLDiffer")>
        <cfset var HtmlSaxDiffOutput =             loader.create("org.outerj.daisy.diff.html.HtmlSaxDiffOutput")>
        <cfset var TextNodeComparator =         loader.create("org.outerj.daisy.diff.html.TextNodeComparator")>
        <cfset var InputSource =                 loader.create("org.xml.sax.InputSource")>
               
        <cfset var finalResult = StringWriter.Init()>
        <cfset var result = TransformerFactoryImpl.Init().newTransformerHandler()>
        <cfset var sr = StreamResult.Init(finalResult)>
        <cfset var prefix = "diff">
        <cfset var cleaner = NekoHtmlParser.Init()>
        <cfset var oldSource = InputSource.Init(StringReader.Init(olderHtml))>
        <cfset var newSource = InputSource.Init(StringReader.Init(newerHtml))>
        <cfset var oldHandler = DomTreeBuilder.Init()>
        <cfset var newHandler = DomTreeBuilder.Init()>
        <cfset var leftComparator = "">
        <cfset var rightComparator = "">
        <cfset var output = "">
        <cfset var differ = "">
        <cfset var diff = "">

        <cfset result.setResult(sr)>
        <cfset result.getTransformer().setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes")>

        <cfset cleaner.parse(oldSource, oldHandler)>
        <cfset leftComparator = TextNodeComparator.Init(oldHandler, Locale.getDefault())>

        <cfset cleaner.parse(newSource, newHandler)>
        <cfset rightComparator = TextNodeComparator.Init(newHandler, Locale.getDefault())>

        <cfset output = HtmlSaxDiffOutput.Init(result,prefix)>
        <cfset differ = HTMLDiffer.Init(output)>
        <cfset differ.diff(leftComparator, rightComparator)>
        <cfset diff = finalResult.toString()>

        <cfreturn diff>
    </cffunction>

</cfcomponent>

Usage:

<cfset var daisy = CreateObject("component","cfc.DaisyDiff").Init(expandPath("../daisydiff-1.1/daisydiff.jar"),"Lighthouse.Utilities.javaloader.JavaLoader")>
<cfset var diff = daisy.diff(olderhtml,newerhtml)>

The result is html that has been marked up by DaisyDiff with special classes.  You can take that and style it in any way that you see fit.

I'm sure there are some refinements that could be done to this CFC.  The class name prefix, for instance, is hardcoded to "diff", and that could be changed if you need to use a different prefix.  Someone more familiar with the Java classes used here could find problems too, which I would welcome.

Comments (2)   Posted on March 29, 2010 4:37:47 PM EDT by David Hammond

MangoBlog is a sweet (ha ha) ColdFusion-based blog, but it doesn't currently support creating multiple blogs using the same codebase.  You can easily make a copy of the code and create another blog that way, but if you're looking at 3, 4, or more blogs, then it's going to start getting out of hand, especially if you want to share the same basic styles for all of the blogs.  (Styles can change, of course!)

One thing that I didn't hear suggested on the MangoBlog forums was to simply create a virtual directory on the webserver in order to create another blog using the same code as an existing blog, and that turned out to work really well.  I wanted to document the steps I took to set that up for the benefit of my future self and others.

Set up Application.cfc

The first thing is to set up Application.cfc to support multiple blogs.  At the top of Application.cfc I added this (my first blog was in a directory called "blog" and all the new ones are in other subdirectories):

this.blogid = ListFirst(cgi.SCRIPT_NAME,"/");
if (this.blogid is "blog"){
    this.blogid = "default";
}

Then I included the blogid in the application name:

this.name = "mango_#right(hash(GetDirectoryFromPath(GetCurrentTemplatePath())),50)#_#this.blogid#_v1_4_2";

Then I used that blogID in the OnApplicationStart:

<cfset facade.setMango(createobject("component",variables.componentsPath & "Mango").init(getDirectoryFromPath(GetCurrentTemplatePath()) & "config.cfm",      
    this.blogid,
    getDirectoryFromPath(getCurrentTemplatePath()))) />

With that initial setup, creating a new blog is a 3 step process:

1. Create new blog and blog author records, copying settings from the default blog

The sql below is what I used

DECLARE @blogid nvarchar(32),@basePath nvarchar(32)
SET @blogid = N'new_blog_subdirectory'
SET @basepath = N'/' + @blogid + N'/'

INSERT INTO BLOG_blog (id, title, description, tagline, skin, url, charset, basePath, plugins, systemplugins)
SELECT @blogid, title, description, tagline, skin, replace(url,'/blog/',@basepath), charset, @basePath, plugins, systemplugins
FROM BLOG_blog
WHERE ID = 'default'

INSERT INTO BLOG_author_blog (author_id,blog_id,role)
SELECT author_id,@blogid,role
FROM BLOG_author_blog
WHERE blog_id = 'default' and role = 'administrator'

2. Copy the blog settings in config.cfm

Make a complete copy of the node that starts "<node name="default">"  and change "default" to your new blog subdirectory name.  Since the config settings for all of my blogs are the same, it would be nice if I could just tell it to use the default configuration, but that doesn't seem to be possible without changes to the MangoBlog code.

3. Create a virtual directory in IIS

Just point the new directory to your original blog directory.  I assume that you can do the same thing in Apache, but I don't have much experience with that.

That's it!

Comments (3)   Posted on March 24, 2010 3:37:42 PM EDT by David Hammond

Lighthouse 3.0 introduces a new task manager that makes setting up and managing scheduled tasks for a website easier and more flexible.

The way it works is that Lighthouse automatically creates a master task in the ColdFusion administrator that runs every 5 minutes.  Whenever this master task runs, it looks for tasks in Lighthouse that are due to run and runs them.  One advantage of this is that no matter how many tasks are set up for a site, there is only one entry in the ColdFusion admin -- which makes managing multiple sites on the same server easier.  More importantly, it allows for more flexible and robust scheduling options than would otherwise be available.

The Lighthouse Task Manager is relatively simple.  Here is the screen to add a task:

Rather than set a url for the task, you must provide a cfc method to run for the task.  So you need to put your task code in a CFC method, and you would then specify the method as "cfc.MyCFC.MyMethod".

You can then specify the interval for the method, any prerequisites (other daily tasks that must be run before the current task is run), and the window of time during which the task should be run.

Most of these options are available in the ColdFusion task manager, but the option to set a prerequisite for a task is an important addition.  This allows you to set up any number of tasks to run at a particular time in the morning, and ensure that they will all run in the correct order, and also ensure that if a task fails, other tasks that depend on it are not run.

The task manager also keeps a log of every time a task is run.  The task method should return a string, which will be logged.  If the task produces an error, the task run will be recorded as unsuccessful, and the error message will be logged.  To log custom error messages, you will want to catch any errors and throw a custom error with the appropriate message.

The task manager is pretty new at this point, and hasn't gone through any trials by fire yet, but I'll be interested to see how it is used as more sites adopt Lighthouse 3.0.

Comments (0)   Posted on February 8, 2010 3:13:49 PM EST by David Hammond

I've run into a strange bug with CFC inheritance.  It definitely seems to be a problem in ColdFusion 8, and I would be interested to see if it's a problem in CF9 also, but I haven't started using CF9 yet.

The problem comes up if you extend a CFC of the same name that is in a different directory.  CFCs in different directories should be treated as completely different components, as far as I know, but something goes wrong if the file name is the same.  For a simple test case, I created 3 CFCs:

/test/Test1.cfc
<cfcomponent output="false">
    <cffunction name="Test" output="true">
        Super Test
    </cffunction>
</cfcomponent>

/Test1.cfc
<cfcomponent extends="test.Test1" output="false">
    <cffunction name="Test" output="true">
        Test
    </cffunction>
</cfcomponent>

/Test2.cfc
<cfcomponent extends="test.Test1" output="false">
    <cffunction name="Test" output="true">
        Test
    </cffunction>
</cfcomponent>

The first file is in a subdirectory, and the other two files are identical except for their names -- one of them is the same name as the first file, and the other one is different.

I created a fourth file with the following code:

<cfset CFC1 = CreateObject("component","test.Test1")>
<cfset CFC2 = CreateObject("component","test.Test2")>
<cfoutput>
<p>#CFC1.Test()#</p>
<p>#CFC2.Test()#</p>
</cfoutput>

What would you expect this file to output?  I would expect:

Test
Test

Instead, I get:

Super Test
Test

In other words, even though /Test1.cfc overrides the Test function, it is ignored in favor of the function in /test/Test1.cfc.  Test2.cfc, on the other hand, works correctly.

My first question, of course, is: Have I missed something?  This sure seems like a glaring bug to me.

Update:
If the Test function is set to access="remote" and called directly from the browser like this:
/Test1.cfc?method=Test
Then it produces "Test" as expected.  All other methods I have tried of calling the function produce "Super Test".  Certainly a head-scratcher.

Comments (1)   Posted on January 25, 2010 8:33:18 PM EST by David Hammond

This post is related to the tokenized auto-complete jQuery plugin available at http://loopj.com/2009/04/25/jquery-plugin-tokenizing-autocomplete-text-entry/.  It is a great plugin.  There are several comments in the discussion on that page regarding the following mods to the plugin. 

• Use of local filtering instead of AJAX.  This makes sense when you have a smaller, pre-defined list of values to select from.
  
  
• Only allowing a token to be selected if it is not already in the list of selected tokens.

It took me a few hours to update the plugin to implement these mods, so I wanted to share the results.

Here is a demo of the plugin modified to for the above items.  Ideally, the plugin would be updated to allow local and remote lookups.  However, I did not have time to implement in this manner, so my version solely works for local lookups.

View Demo

Comments (0)   Posted on November 4, 2009 1:26:58 PM EST by David Hammond