A couple of days ago, I put up a post about the SANS/CWE list of 2009's Top 25 Most Dangerous Programming Errors, and noted I would cover some of the items individually.

Today, I will briefly cover a couple that are somewhat related, in regards to the title of this entry. What I hope to get across is this; When you release an application (desktop or Web/Intranet), a programmer must always remember that malicious users will get access to the client side code. As a result, they can - and will - exploit whatever holes have been left. My goal with this article is to add something to the 'blogosphere that describes why these issues are important for programmers to consider at all times. If you are not a programmer, and never intend to be, perhaps this will be rather boring! Heck - even if you are, it may still be boring.

So the items which apply here are; Improper Input Validation, Improper Encoding or Escaping of Output, Failure to Preserve SQL Query Structure (aka 'SQL Injection') and Error Message Information Leak. All of these errors create a gold mine for people who want to use your hard work for their own nefarious purposes. These are very common and easily-made errors by inexperienced programmers, especially because these errors can be made in relatively simple web site code, where less-experienced programmers might more often find themselves creating something that will be available to be accessed by those seeking 'low-hanging fruit' to exploit. A common misconception is, "I'm not exactly creating another Amazon.com or a national bank's online-banking system; Who is going to try to hack my web site?" The answer to that question; Anyone looking for a group of users to exploit, a web server to send e-mail spam from, or a free place to host illegal content.

More...

The Common Weakness Enumeration (CWE) and SANS Institute (SANS) have posted the 2009 Top 25 Most Dangerous Programming Errors. As a developer who has always written code meant to be used in traditionally 'hostile' environments, it sometimes surprises me to see some of these items continue to be listed, and to continue to happen so frequently. Of course, I must admit that I probably have no place passing judgment on programmers in whose situations I have never quite been. I have never worked in a very large development shop with practically unrealistic deadlines set by management/marketing types, nor have I been in a project management position on any large-scale open-source project. Large-scale projects - whether open-source or wholly commercial - tend to have their own (often different) pressures which I absolutely can understand would lead to a desire to take shortcuts, and sometimes there is no true alternative than to work quickly, when one may prefer to work carefully.

However, I also do not want to provide excuses. Persons who write code for software are intelligent folks, by and large. They should know - or learn quickly - the many programming tricks and standards which can help them avoid these mistakes. Those who are tasked with dealing directly with management should also learn the intra-personal skills needed to assure that their teams have the time they need to develop code that is reasonably safe. One reason CWE and SANS have posted this list is to help developers hone in on these errors, and devote the time and energy to assuring that these issues do not become their issues. Software development is an art to a degree, and it is such a complex one that mistakes are probably a given in all but the simplest projects. But knowing what some of the most commonly-made and -exploited errors are should help.

As an academic exercise, I am going to cover some of these items in some upcoming entries - including noting some cases where I have struggled with them. For now, I just wanted to pass along the link to the very useful list itself.

I came across a somewhat obscure issue that, it seems, really should not be so obscure. I am working on an application that will be used to display rotating videos, a 'ticker' and rotating graphics in lobbies/waiting areas, on the client's hardware. I am using Windows Forms 2.0, DirectX 10 to display the videos, and a simple WinForms PictureBox control to display the graphic images. My intent was that the 1.0 version of the program would no longer use the PictureBox in order to support animated GIFs to be among the graphic items. My prior experience was that simply using a PictureBox.Image property would not properly display an animated GIF, but only show the first frame.

So, I read somewhere that with WinForms 2.0, that was no longer the case. So, I figured I'd try one out before going too far. I put an AniGif in the rotation, and it loads up. I wait the 5-seconds I know this particular GIF has until frame 2 comes up, and, lo and behold, an uncatchable exception: A generic error occurred in GDI+

Taking a peek at the stack trace from the exception, I find that there are, in fact, WinForms methods being called on my behalf at the appropriate time, trying to display the next frame of the image. But they are not working. "Generic Error" and all. How helpful...

More...

I got my start actually making a 'living' as a software developer massaging data; Getting something that was an old mess into a more modern, accessible format that could be used into the future.

At the time, I worked for the federal government, and I was charged with taking what amounted to a huge, manually-edited spreadsheet and normalizing the data so that it could be imported into a recently designed Oracle database. In actuality, that project was sort of meant to be busy work, and it was expected that I would spend a few months manually scanning tens-of-thousands of records to do things like assure the address fields were split into their parts (line 1, line 2, city, state, postal code, country). What I actually ended up doing was spending a few minutes starting to do that, and then realizing that there really were some patterns to the data which could be identified by a set of rules for at least 99% of the records.

So, I spent a couple days playing with some parsing code. The office I was in was all Microsoft on the desktop, so I created a simple Access 97 database the schema of which duplicated the target Oracle db, and then began to write some Visual Basic code within Office to parse every line in the spreadsheet that could be parsed, and slap it in. In the end, I was left with perhaps 1000 records which could not be readily parsed by code and needed to be manually converted. The whole job took about two weeks, largely because I had not previously used Visual Basic within Office at all before then, and so needed to learn a bit of the environment there.

What's That Got to Do With Today?

What's fitting, then, is that my first real post here will also be about moving some data from a manual format into a normalized database. I am working on a project for the Volunteer Action Center of the Mid-Ohio Valley (VACMOV), to take their online listing of programs and agencies and put it into a format with which they can more readily work. Right now, it is simply in individual, plain HTML files. Each agency/program has its own, and there are nearly 800 of them. These records are found on the current web site by key words, of which there are just over 400. Each key word itself has a manually-edited HTML page, which lists and links each record that applies. As one might expect of such a setup, editing this data is quite difficult to do. Adding a keyword to a record, adding a completely new keyword, adding a whole new agency/program record, and even just editing a program's information are all much more complex than they need to be. Of course, updating the appearance of the pages also is effectively impossible, since there are well over 1200 pages which would need to be edited.

You can see the VACMOV's Information and Referral pages to see the current system.

At this stage, I have been able to get all but 13 of the current records to be parsed into a database. The HTML files of those 13 records can be altered to parse except two, which will need to be manually entered into the new system. This was by far the hard part; Having been written a while ago, the HTML files do not follow XHTML standards, but rather the more lax, original HTML standards. For example, there are <p> and <li> tags throughout without their matching closing tags. Luckily, though, this has not been a big impediment. I've been able to use Regular Expression pattern matching to locate individual fields, often by their labels, which are surprisingly uniform.

So, next I will be writing a parser to get all of the key words in. Each key word HTML file will be scanned, and every matching record will have the appropriate key word added to it in the database. After that comes the generally much simpler task of presenting the information. We'll be hosting the VACMOV's web site soon, and providing the system to run the Information and Referral portion. I am not sure that the current method of clicking a letter of the alphabet and then scanning for a keyword in that list is the best way for people to find what they need, so I will instead be providing a search form which searches both the keywords and the text fields of the records themselves. I will leave something that looks a lot like what they have now with the alphabetical thing there only so that search engines will continue to have clickable links by which to find the records in their new locations, so they will be indexed there too. Since the new database also happens to store the old HTML file name of the records, I will also put in place a 404 error handler which detects if the page requested was one of those old records, and redirects them to the new one if so.

Volunteer Action Center

The Volunteer Action Center is a local non-profit group in our area which helps agencies find what they need to provide the services they do. We publish "Get Connected" from the VAC every month in Out & About, a Family Guide, and we are happy to donate our services for their web site here as well.