Modern Signal Blog

Topics

News, tips, tricks and discussions related to web strategy, web design, and custom web application and database development.

Viewing posts for topic: "ColdFusion". View all posts.

ColdFusion Rate Limiter

I was recently looking for a way to throttle the number of requests to a site from bad bots that sometimes flood a server with dozens of requests in a few seconds. I found a very nice simple function by Charlie Arehart:
http://www.carehart.org/blog/client/index.cfm/2010/5/21/throttling_by_ip_address

Before proceeding, you might want to look over Charlie's article, which nicely lays out many of the considerations and caveats that come with limiting the number of requests by IP address.

Aside from all the caveats, Charlie's function had a couple things I wanted to adjust. One was that, as written, it would block even "good" bots, like Googlebot, if they made requests more frequently than the "duration" parameter. For instance, if you set it up to allow up to 6 requests every 3 seconds (as I did), it would end up blocking Googlebot from making a steady stream of requests at the frequency of once per second. So even though Googlebot was only making 3 requests every 3 seconds it was getting blocked.

Adjusting Charlie's function to stop doing that was very easy. But the other thing I was concerned about was whether keeping a list of IP addresses in an application variable would scale very well. How long would the list have to be before it would affect performance? I didn't really know, but felt it would be better to have a solution where the list would have old IP addresses pruned periodically. I toyed with the idea of using a scheduled task to go through the list and removed old entries. That seemed cumbersome, though. In the end, since the site I was working on was running ColdFusion 9, I thought it would be a good opportunity to use ColdFusion 9's new caching functions. Using ColdFusion's built-in caching functions means the I can have old IP addresses cleaned up automatically.

So here is my version of the rate limiter. This is generally more forgiving than Charlie's version of the function. In Charlie's version if a bot is blocked but continues to make requests (e.g. Googlebot), it will continue to be blocked until it pauses for the "duration". In my version, the bot is blocked for the "duration" and then allowed to make more requests before being blocked again. My main goal is to block the rogue bots that flood the system with 10 or 20 requests a second for a short time, and this does a pretty good job of that.

<cffunction name="limiter">
 <cfargument name="duration" type="numeric" default=3>
 <cfargument name="count" type="numeric" default=6>
 <cfset var cacheId = "rate_limiter_" & CGI.REMOTE_ADDR>
 <cfset var rate = cacheGet(cacheId)>

 <cfif isNull(rate)>
 
 <cfset cachePut(cacheID, {attempts = 1, start = Now()}, createTimeSpan(0,0,1,0))>
 <cfelseif DateDiff("s", rate.start, Now()) LT arguments.duration>
 <cfif rate.attempts gte arguments.count>
 <cfoutput>
 You are making too many requests too fast,
 please slow down and wait #arguments.duration# seconds
 </cfoutput>
 <cfheader statuscode="503" statustext="Service Unavailable">
 <cfheader name="Retry-After" value="#arguments.duration#">
 <cflog file="limiter" text="#cgi.remote_addr# #rate.attempts# #cgi.request_method# #cgi.SCRIPT_NAME# #cgi.QUERY_STRING# #cgi.http_user_agent# #rate.start#">
 <cfif rate.attempts is arguments.count>
 
 <cfset cachePut(cacheID, {attempts = rate.attempts + 1, start = Now()}, createTimeSpan(0,0,1,0))>
 </cfif>
 <cfabort>
 <cfelse>
 
 <cfset cachePut(cacheID, {attempts = rate.attempts + 1, start = rate.start}, createTimeSpan(0,0,1,0))>
 </cfif>
 <cfelse>
 
 <cfset cachePut(cacheID, {attempts = 1, start = Now()}, createTimeSpan(0,0,1,0))>
 </cfif>
</cffunction>

Comments (4) Posted on April 2, 2012 2:58:45 PM EDT by David Hammond

Reporting with SQL PIVOT

I wanted to write about another bit of code that I created for the What Color Is...? website I talked about in my previous post.

Displaying the Results

Probably the most interesting and challenging part of this project was to come up with a way to generate and display the results in an efficient way. Colors were all saved in the database as 3 separate fields (Red, Green, and Blue). This made it easy to select a color average for all responses. Here is the SQL:

SELECT t.name ,
        COUNT(*) AS Count,
        AVG(IsNull(red,255)) as Red ,
        AVG(IsNull(green,255)) as Green ,
        AVG(IsNull(blue,255)) as Blue
FROM    WCIResponse r
        INNER JOIN WCIThing t ON r.ThingID = t.ThingID
GROUP BY t.name

One of the options in the survey is to select "No Color", which is saved in the database as null values. These are converted to 255 so that it is the same as if they selected white.

That was pretty easy, but I also collect some basic, anonymous demographic information to enable breaking down the results a bit. I ask for age, for instance. What I wanted to do was to break down the results by age group, i.e. 20-29, 30-39, etc. This is more complicated, of course, and I didn't want to get into doing a massive number of queries and loops, etc, to build up the results. Fortunately I have recently gotten to know more about the SQL Server statement PIVOT. PIVOT is made for creating just such reports as this. This is somewhat more complicated than a simple PIVOT, because I don't want a column for every age, but I want to group the ages by decade.

I'll run through the ColdFusion/SQL code I used to make this work.

First, I set a variable with an SQL expression to create the name of the age group, e.g. "40-49":

I only want to include the age ranges for which I actually have data, so I do an initial query to get the age ranges in the data. Note that the AgeRange is wrapped in brackets so that it can be used as the column name in the next query, e.g. "[40-49]".

<cfquery name="GetAges" datasource="#application.dsn#">
 SELECT DISTINCT
 '[' + #preserveSingleQuotes(AgeRange)# + ']' AS AgeRange
 FROM WCIPerson
 WHERE Age IS NOT NULL
 ORDER BY AgeRange
</cfquery>

I can then build the PIVOT query including all of the age ranges. I'm still not comfortable with the PIVOT syntax. I've only used it a few times, and it's still hard for me to wrap my head around without looking at documentation and examples. If you're new to PIVOT too, this site has a few good examples. As you can see below, I reuse the AgeRange variable I set above twice.

<cfquery name="GetResults" datasource="#application.dsn#">
 SELECT t2.name
 <cfloop query="GetAges">
 ,ISNULL(t2.#GetAges.AgeRange#,'0,255,255,255') AS #GetAges.AgeRange#
 </cfloop>
 FROM ( SELECT t.ThingID ,
 t.name ,
 #preserveSingleQuotes(AgeRange)# AS AgeRange,
 CONVERT(VARCHAR, COUNT(*)) + ','
 + CONVERT(VARCHAR, AVG(ISNULL(red, 255))) + ','
 + CONVERT(VARCHAR, AVG(ISNULL(green, 255))) + ','
 + CONVERT(VARCHAR, AVG(ISNULL(blue, 255))) AS Color
 FROM WCIResponse r
 INNER JOIN WCIThing t ON r.ThingID = t.ThingID
 INNER JOIN WCIPerson p ON r.PersonID = p.PersonID
 WHERE p.Age IS NOT NULL
 GROUP BY t.ThingID ,
 t.name ,
 #preserveSingleQuotes(AgeRange)#
 ) AS t PIVOT ( MAX(color) FOR AgeRange IN ( #ValueList(GetAges.AgeRange)# ) ) AS t2
</cfquery>

The value returned here for each age range is actually a comma-delimited list of values, where the first value is the number of responses, and the next three are the RGB values. When the data is pulled into the application (as JSON), I am able to parse these values and fill a table to colored blocks to show the selected colors for each age range.

Although the What Color Is...? site doesn't have a lot of data at this point, I believe this code should scale pretty well. Of course in the unlikely event that the site becomes an internet phenomenon and gets millions of responses I may have to revisit this code.

Comments (0) Posted on January 31, 2011 5:09:23 PM EST by Catherine Field

Setting Request Limit on IIS7

We had a client trying to upload a 44mb file, and it was failing. The ColdFusion page that that did the upload had a very high timeout setting, so I didn't think that timeout was the issue. The ColdFusion administrator had a request limit set of 100MB (which I believe is the default). What was up?

It took a bit of googling to realize that IIS7 (the webserver on our Windows 2008 Server), had a default request limit of 30MB. The solution is rather simple. With IIS7, a web.config file can be used to set a variety of webserver settings, so I added one to the site that was having the problem. Here are the contents of the file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <system.webServer>
 <security>
 <requestFiltering>
 <requestLimits maxAllowedContentLength="1024000000"/>
 </requestFiltering>
 </security>
 </system.webServer>
</configuration>

This sets the request size limit to 100MB, to bring it in line with ColdFusion. This isn't the only thing that can be controlled with web.config. For some information on how to tighten security on a site using web.config see this page: http://www.petefreitag.com/item/741.cfm

Comments (1) Posted on May 3, 2010 1:44:47 PM EDT by David Hammond

Simple DaisyDiff CFC Wrapper

Some background first...

I had the need to do diff of html content in a project I was working on, which brought me pretty quickly to DaisyDiff, a really nice Java-based utility. DaisyDiff doesn't however, have a simple built-in function to do a diff of two strings. There is a command-line option, which takes the paths of two files as arguments, and also a java api that take a number of java objects as arguments. What I wanted was a function that took two strings and output the results, but DaisyDiff has no such simple function.

I don't really do java development -- that is I've done some in the past but it's been a while and it would probably take me some time to get my development environment up to snuff. Besides, I didn't really feel like dealing with compiled code.

A quick google search, of course, turns up CFX_CompareHTML and the JavaLoader version of the same thing. So I used that, and it worked fine. But it was using an old version of DaisyDiff, and it seemed to have some bugs with UTF characters and such. What I really wanted to do was to use JavaLoader to load the current version of DaisyDiff. After much stumbling around in the code, I found that the test suite in the DaisyDiff repository has exactly the function I wanted -- it compares two strings and returns the result.

So, long story short, I took the code from that function and pulled it into a CFC, using JavaLoader, and rewrote everything in CFML. The result is the simple function I was after.

So anyway, here it is:

Usage:

The result is html that has been marked up by DaisyDiff with special classes. You can take that and style it in any way that you see fit.

I'm sure there are some refinements that could be done to this CFC. The class name prefix, for instance, is hardcoded to "diff", and that could be changed if you need to use a different prefix. Someone more familiar with the Java classes used here could find problems too, which I would welcome.

Comments (3) Posted on March 29, 2010 4:37:47 PM EDT by David Hammond

Setting Up Multiple MangoBlog Instances with Shared Code and Database

MangoBlog is a sweet (ha ha) ColdFusion-based blog, but it doesn't currently support creating multiple blogs using the same codebase. You can easily make a copy of the code and create another blog that way, but if you're looking at 3, 4, or more blogs, then it's going to start getting out of hand, especially if you want to share the same basic styles for all of the blogs. (Styles can change, of course!)

One thing that I didn't hear suggested on the MangoBlog forums was to simply create a virtual directory on the webserver in order to create another blog using the same code as an existing blog, and that turned out to work really well. I wanted to document the steps I took to set that up for the benefit of my future self and others.

Set up Application.cfc

The first thing is to set up Application.cfc to support multiple blogs. At the top of Application.cfc I added this (my first blog was in a directory called "blog" and all the new ones are in other subdirectories):

this.blogid = ListFirst(cgi.SCRIPT_NAME,"/");
if (this.blogid is "blog"){
this.blogid = "default";
}

Then I included the blogid in the application name:

this.name = "mango_#right(hash(GetDirectoryFromPath(GetCurrentTemplatePath())),50)#_#this.blogid#_v1_4_2";

Then I used that blogID in the OnApplicationStart:

With that initial setup, creating a new blog is a 3 step process:

1. Create new blog and blog author records, copying settings from the default blog

The sql below is what I used

DECLARE @blogid nvarchar(32),@basePath nvarchar(32)
SET @blogid = N'new_blog_subdirectory'
SET @basepath = N'/' + @blogid + N'/'

INSERT INTO BLOG_blog (id, title, description, tagline, skin, url, charset, basePath, plugins, systemplugins)
SELECT @blogid, title, description, tagline, skin, replace(url,'/blog/',@basepath), charset, @basePath, plugins, systemplugins
FROM BLOG_blog
WHERE ID = 'default'

INSERT INTO BLOG_author_blog (author_id,blog_id,role)
SELECT author_id,@blogid,role
FROM BLOG_author_blog
WHERE blog_id = 'default' and role = 'administrator'

2. Copy the blog settings in config.cfm

Make a complete copy of the node that starts "<node name="default">" and change "default" to your new blog subdirectory name. Since the config settings for all of my blogs are the same, it would be nice if I could just tell it to use the default configuration, but that doesn't seem to be possible without changes to the MangoBlog code.

3. Create a virtual directory in IIS

Just point the new directory to your original blog directory. I assume that you can do the same thing in Apache, but I don't have much experience with that.

That's it!

Comments (3) Posted on March 24, 2010 3:37:42 PM EDT by Catherine Field

ColdFusion CFC Inheritance Bug

I've run into a strange bug with CFC inheritance. It definitely seems to be a problem in ColdFusion 8, and I would be interested to see if it's a problem in CF9 also, but I haven't started using CF9 yet.

The problem comes up if you extend a CFC of the same name that is in a different directory. CFCs in different directories should be treated as completely different components, as far as I know, but something goes wrong if the file name is the same. For a simple test case, I created 3 CFCs:

/test/Test1.cfc
<cfcomponent output="false">
 <cffunction name="Test" output="true">
 Super Test
 </cffunction>
</cfcomponent>

/Test1.cfc
<cfcomponent extends="test.Test1" output="false">
 <cffunction name="Test" output="true">
 Test
 </cffunction>
</cfcomponent>

/Test2.cfc
<cfcomponent extends="test.Test1" output="false">
 <cffunction name="Test" output="true">
 Test
 </cffunction>
</cfcomponent>

The first file is in a subdirectory, and the other two files are identical except for their names -- one of them is the same name as the first file, and the other one is different.

I created a fourth file with the following code:

What would you expect this file to output? I would expect:

Test
Test

Instead, I get:

Super Test
Test

In other words, even though /Test1.cfc overrides the Test function, it is ignored in favor of the function in /test/Test1.cfc. Test2.cfc, on the other hand, works correctly.

My first question, of course, is: Have I missed something? This sure seems like a glaring bug to me.

Update:
If the Test function is set to access="remote" and called directly from the browser like this:
/Test1.cfc?method=Test
Then it produces "Test" as expected. All other methods I have tried of calling the function produce "Super Test". Certainly a head-scratcher.

Comments (1) Posted on January 25, 2010 8:33:18 PM EST by Catherine Field

ColdFusion Function to Set HttpOnly Cookie

It's gotten so I can't think about writing any code without fretting about security while I'm doing it. I know, I know, that's a good thing. Still one does long for the old days when the makers of small to medium-sized websites could get away with lax security just because the big sites had enough security holes to keep the hackers busy. And those hackers didn't have as many automated tools to make hacking lots of sites easier.

Anyway, those days are long gone and none of us has to be labeled paranoid, because they really are out to get us.

So, on my security checklist was to start using HttpOnly cookies for potentially sensitive cookies (like those related to login). HttpOnly cookies have been supported for a while by Internet Explorer, and Firefox has recently started supporting them. What's good about HttpOnly cookies is that their values can not be accessed through javascript, which stops a variety of cross-site-scripting attacks. For more info, here is a good page:

http://www.12robots.com/index.cfm/2009/1/5/mmmmMMmmmmmmm-Cookies--Security-Series-12

The problem that comes up immediately for a ColdFusion developer is that the CFCOOKIE tag does not support httponly cookies. This seems like a gross oversite, especially since it should be very easy to implement. In order to use httponly cookies, you need to use the CFHEADER tag to to write the specially formatted cookie header to the browser. I looked around but couldn't find anyone online that had created a function that handled all of the functions of CFCOOKIE with the addition of httponly. Anyway, here is what I came up with.

It's actually pretty simple, with the expires portion of the header being the only thing that required a little work. I have attempted to mimic the CFCOOKIE functionality so that this can be used more or less as a simple replacement for that tag. Here's an example of usage:

I hope this proves helpful to others.

Comments (2) Posted on October 8, 2009 3:13:19 PM EDT by David Hammond

Disabling Script Execution in a Directory in IIS7

Last night I was trying to disable script execution in the uploads folder of a site running in IIS (Windows 2008). It is also running ColdFusion, which turned out to be important. I ran into a couple problems.

The first was that I had no idea how to do that in IIS7. I knew how to do it in IIS6, but everything is different in IIS7. I thought I would be able to just find it, but after poking around for a while, I gave up and ran to Google. Turns out the new location to set this is in Handler Mappings. If you go to the Handler Mappings feature for a directory and click on "Edit Feature Permissions...", you can uncheck the "Script" permission.

So I did that, and I thought I was done, but then I noticed that not only could I not run scripts from the directory, but trying to access a static file, such as a gif, also gave me a "403 Access Denied" message. Strange.

It turns out that ColdFusion installs a wildcard script map, which means that it is set up to handle all files, even static ones. I don't know what the reason for this is (and I would love to find out if anybody knows), but it was getting in the way of the default static file handler. I had to remove that handler for the directory in order for the static files to be served properly.

Comments (5) Posted on July 28, 2009 11:14:36 AM EDT by David Hammond

Testimonials

I love working with Modern Signal! Their CMS is very easy to use and they are incredibly responsive to questions or challenges I bring them.

- NALP
Modern Signal worked with us to understand our needs and figure out what solution would work best for us. Our Lighthouse CMS is perfectly suited to our website goals. When we later needed to modify the CMS, they again took the time to understand exactly what was needed and then built that functionality rather than delivering a cookie cutter solution.

- Ecosystem Investment Partners
This was by far the smoothest website redevelopment I have ever experienced. Modern Signal was a wonderful company to work with and we greatly value our working relationship.

- National Association of Student Financial Aid Administrators
Modern Signal has a professional staff that was very responsive to our needs during all phases - scoping, developing, implementing and maintaining - of our project. We have been pleased with their ability to deliver quality work on time and on budget. If given the opportunity, I would work with them again.

- The National Center for Safe Routes to School
Modern Signal understands our business - from future needs to current limitations - so their solutions are always scalable, solid, and service-oriented.

- National Association of Home Builders
Modern Signal significantly enhanced our site to be more efficient and user-friendly. They provide excellent customer service with timely and cost-effective solutions.

- Center for Medicare Education
Modern Signal has been a great partner for us for over the past 10 years. As our business grew and our needs changed, Modern Signal was able to work with us to adjust our website platform in the ever-changing online world. Their service and response level has been second to none, and we've been never been happier with our relationship with them.

- Charm City Run
I felt as if my company was their only client. They responded to my needs quickly and efficiently despite short turn around time and intense demands.

- Teaching Strategies, Inc.
We wouldn’t have gotten where we are today without your support over the years. Modern Signal has always been a great partner to us.

- Kirk Gillis, Managing Director at Zoom Tanzania