I have a love/hate dislike/hate relationship with web statistics report generation tools. I'm using AWStats now, and it works fine enough. But very often, my clients want to know, "how many times has X page been viewed?" and "where are people being referred to Y page from?" These questions can not always be readily answered by a full stats reporting system.

A previous practice I used for a while was to import all the log data into an SQL database. I figured from there I could run whatever SQL queries I needed to learn whatever anyone needed to know - I could even whiz-bang-wow a client by doing it right at a meeting! So I never even tried to do that, but I did use it to build some simple reports for some clients on specific, targeted pages or areas of their site. But it always seemed like too much effort, especially for some quick, one-off questions a client might have.

Linux Shell Commands to the Rescue

So I thought I'd try something to see how it works. For the benefit of AWStats, I already had a consolidated, ordered log file of the entire time period in question in plain text, W3C format. This is just perfect for using the *nix shell tools on, isn't it? After SCPing the file to my local Linux box, I ran this series of commands, piping each to a new file.

grep

First, I grep'ped for the specific page I was looking for. The site in question runs ASP.NET and I'm using URL Rewriting, but as far as the logs are concerned, the page being requested is the 'real' one, complete with whatever ugly URL applies. So my grep argument was this: 'GET /Pages.aspx id=70' The file that resulted contained only log lines that were requests for that specific page. I then ran the file through some grep -v to quickly filter out results I knew we did not want, such as hits from my IP, the client's IP, and hits with the back-end management interface as the referrer.

cut

Once I was sure I had a file containing only the 'hits' on the page I wanted, I used cut -d' ' -f8 and piped that to a new file. This cut column 8 (which happens to be the referrer in my case) from the file, using the space as the delimiter. By default, cut works on characters, not delimited fields. the -d option lets you specify a field delimiter. I had to specify the delimiter in single quotes since I was using a space. For comparison, if the delimiter had been a colon, the command would have looked like this:  cut -d: -f8 Note that there is no space between the -d option and its argument.

sort and uniq

Once I had a file containing nothing but the HTTP referrers of the page in question, I wanted a list of the referrers and their frequency, sorted from most to least frequent. Just a little bit of piping gets me there:

cat INPUT_FILE | sort | uniq -c | sort -nr > OUTPUT_FILE

Easy enough, eh? So what I am doing is easy enough to follow, really; my list of referrers first needs to be sorted (I could have used the -f option to sort to ignore case, but I did not need to, and you may not want to, actually) then it is piped in to uniq -c, which produces a list of only each unique line in the file and, thanks to the -c option, precedes each line with a count of how many times it exists. Next I need to sort it again; this time I am sorting it in reverse order (-r) and numerically (-n) and sending it to the final file, which contains exactly what the client wants.

Conclusion

Depending on what information is needed, the above can be a quicker way to produce it than other methods. I work with SQL day-in, day-out, so it was natural for me to initially think of that as the first solution. However, one thing I like about this is that I can work with it more 'on the fly'; at each stage I have an output file narrowed down appropriately and I can think of the information I want in a more step-by-step way.

If someone is sending you any Encapsulated PostScript (EPS) files via e-mail, I'd definitely recommend that they be sent in some form of an archive. Not only is the compression likely to be useful, but it can also prevent them from being corrupted. EPS files are basically plain text, with some blocks of encoded byte values mixed in. For what ever reason, the process of e-mailing these documents can corrupt them; adding garbage to the beginning and/or end of the file. Perhaps it happens while being encoded/decoded by e-mail clients, or maybe in the process of being scanned for viruses. It does not happen to everyone, though... if it happens to you, and you can't get your source to quickly send you a replacement, here's a way to try to fix the file yourself.

First, be sure you have a backup of the file. Of course! Open the EPS file in a text editor. It should appear to start with some neatly ordered lines similar to this:

%!PS-Adobe-3.1 EPSF-3.0
%ADO_DSC_Encoding: MacOS Roman
%%Title: File-Title.eps
%%Creator: Adobe Illustrator(R) 13.0
%%For: Andrew Barber
%%CreationDate: 4/26/10
%%BoundingBox: 0 0 143 57
%%HiResBoundingBox: 0 0 142.4434 56.6699
%%CropBox: 0 0 142.4434 56.6699
...snip...

If you do not see anything like the above toward the beginning of the file, stop right now; something else is probably going on, and this procedure won't help you. However, what you may see is a line or two of apparent garbage preceding the "%!PS-Adobe-3.1 EPSF-3.0" line. The garbage might be on the same line as that correct starting line, so you may need to scroll to the right to find it. Delete all of the garbage before the percent sign.

Next, scroll down to the end of the file. You will see lots of code lines, likely some encoded blocks of binary data - but it should all be well formatted either as code or blocks of data. The very last line of the file should read %AI9_PrivateDataEnd, followed by a single line break. However, you might again see garbage starting on that line after. Delete all the garbage at the end, but leave that single empty line after the %AI9_PrivateDataEnd.

Hopefully that should do it; save the file, and try to load it again.

I do not very frequently work with video, but occasionally enough that it's a headache at times. I have a custom lobby monitor software program I created for a client, which displays videos, text information, and graphical promotions to clients waiting in their lobby on multiple high-def monitors they had installed. Rather than pay premium prices for the hardware, and then somewhat excessive monthly subscription fees for ongoing service for similar systems, they selected their own hardware, had a local A/V provider get the audio and video hooked up, and had me write software to operate the system. They ended up paying about the same up-front overall, but are saving considerable fees monthly.

Of course, it's important to keep the content fresh, and the videos are especially an issue, due to the greater effort to create such content. They use television commercials they have had produced, videos available to them from trade associations and partners, and other content they find online which is appropriate for their customers, and for which they have (or can get) appropriate permission.

The Problem

The system I created for them is capable of playing a great many formats using the DivX codecs. It can even take the .vob files directly from a DVD and use those for playback, but those are typically fairly large files, and it would be easier to work with them after being converted to, say, a more approrpriately-sized WMV. There are also some formats that can't be played with what we've got, and I would rather not introduce extra complexity to the system itself to try to get them to work.

What I needed, then, was something that could convert videos from pretty much any format out there, to any of a few specific formats that are best used on the system. It needed to be something simple enough that the client could use it, if need be... but even I need something simple and to-the-point, so as not to waste time trying to remember which options and settings and etc. to have to use every time I needed to convert one MOV file for them. I'd had lots of trouble in the past with the output videos not being playable, and I could never figure out if it was a problem with the converter being used, the computer, or just stupid settings on my part.

The Solution

So I finally found a solution that seems to work for all of my needs; WinFF is a free (as in 'beer' and 'speech') GUI for the command-line FFMPEG conversion software, which it includes as a compiled program as well. It allows you to specify any number of input video files (which can be of various formats/file types), and allows you to specify a pre-set output format and options, or to fine-tune the settings you need, if you know what you are doing (I don't!). Tell it where you want output to go, and set it loose; a command-prompt window opens and you are shown the progress from FFMPEG as it converts the files.

I have had no trouble with conversions performed with WinFF/FFMPEG as of yet. In particular, the large MOV and VOB files that had caused me trouble with other converters worked like a charm.

There are downloads available for Windows and Debian, Ubuntu and Red Hat-based Linux distributions, as well as source code.

In talking with various business relationships I have who have gotten into social networking sites like Facebook recently, one common series of complaints involves all of the 'fluff' that can be found there. Invites to multiple games each day; their home page feed littered with updates from these same games, or other apps they simply are not interested in. One worry of mine, personally is that these complaints seem to be of the, "if I can't figure out this stuff, I'll probably stop using Social Networking altogether" variety.

So I've been planning a simple blog post to detail some of the tools Facebook provides to help here. But then today, I came across a Blog challenge from Tim Ford, otherwise known as the SQL Agent Man (link). The Blog entry was titled, "Whose Blog Is It Anyway?" (link) and the challenge is inspired by the game "Scenes From a Hat" from the improv show "Whose Line Is It Anyway?". I considered starting over and writing something else but then I thought to myself, I've started, so I'll finish!

More...

Regionerate is an open-source add-in for Visual Studio which enables developers to easily apply code layout rules in C# source. I've been using it for a few months now, and have found it to be incredibly useful, with only two issues; one not really even worth mentioning, but the other could be a problem for some.

First, though; what it does! From the Regionerate Introduction page:

Regionerate is an automated tool which helps you to preserve your code's layout over time. Regionerate lets you define regions in your code and determine the way members (fields, methods, properties etc.) should be placed inside them.

Regionerate is a zero-friction tool - when setting up, you can choose a Code Layout (the way you want your code to look) or just use the default Code Layout. From that moment on, Regionerate will make sure your code follows that Code Layout.

I got my grubby paws on a brand-new Verizon HTC Ozone at the local Alltel-turned-Verizon store on Friday, three days before the official brick-and-mortar store release date of tomorrow. It has been for sale online since the 29th, and we're not exactly talking about an Apple iPhone or Palm Pre here, so it just took a small amount of whining to get it. The Verizon store being swamped with folks taking advantage of the special deals to switch their Alltel accounts to Verizon (our situation) probably helped, too.

The Ozone is a down to business, Windows Mobile 6.1 smartphone made for those who, like me, might have been considering something like a Blackberry Pearl for Verizon's 3G EVDO rev A network. However, it's known as a 'World Phone', which apparently means that it also works on GSM, GPRS and EDGE networks. Also radio-related; It is Wi-Fi enabled, so you can connect to a wireless computer network (802.11b/g) if, for example, the EVDO signal is not available (or if you happen not to have a data plan).

More...

Thanks to a Facebook link my friend Bev Shaw posted, I recently found SimplyNoise.com. This site has a Flash-based 'White Noise' generator which actually goes beyond white noise, to also include pink and red/Brownian noise. You can find a link on Simply Noise to a wikipedia article on what white, pink and red noise are technically speaking. From a simple perspective, though, they can help drown out the noise of daily living to help you concentrate, relax, or just clear your mind. There are desktop programs which can do this, as well as inexpensive 'noise machines' you can purchase, but I've not found anything that I like as much as Simply Noise.

However, I don't like that it's a web page. I like to keep my desktop as clean as possible while working; particularly while working on the things where I would most want to have a white noise generator running. The fewer things are on my task bar, the better. It would be a hard habit for me to break to stop closing the last Firefox instance when I'm done checking e-mail, news, or what have you. The first couple weeks of using Simply Noise, I found this to be a problem.

More...

This is pure troll-baiting, but I've long worked (struggled?) in a mixed Mac/PC world, and the best thing I love about Macs anymore is that, well - they are PCs. Although the price tag certainly reflects the features, my 2-year old MacBook Pro (MBP) has a dual-core, 64-bit CPU, has 4gigs of DDR2, and has the best in USB and Firewire connectivity. I fit easily into any cliquish computer-user group; Mac lovers across the cafe smile warmly at the sight of the softly glowing apple on the back of my screen, while I can share a conspiratorial chuckle with the PC-loving nerds as I boot up into Vista Ultimate.

The Problem

When sitting at my desk, with my external peripherals arrayed around me, I'm in heaven. Nothing is lacking, and there would never be any indication that I'm using anything other than a fully-equipped PC. Those who have gone back and forth between PCs and Macs know that the keyboards are not exactly the same. But my Microsoft keyboard works beautifully while my computer runs as a Mac or PC, with subtle, secondary icons on the keys to note how they work for the Apple operating system.

More...

MyMobileR is a Windows application which allows you to view and interact with your Windows Mobile-based (versions 2003, 5 and 6) smart phone or pocket PC, via ActiveSync.

I keep my phone in its USB dock when I am at the compter to keep the battery topped off, and so I always know where the phone is when I'm ready to rush out the door at the very last minute. I use the dial code for call forwarding (*72[phone number], on Alltel anyway) to set my phone's calls to forward to my location, and even have speed-dials set to forward my cell to home or office, and one to stop forwarding (*73). Since my contacts, calendar, to-dos, notes and documents are all automatically synced, this works perfectly for me, with one big exception: text messages.

MyMobileR is freeware, and is a fairly simple application in scope; It simply provides an interface to your phone. I can type with the keyboard, use the mouse to click/drag instead of the stylus (or finger, for touch screens), and it even includes function key bindings to send the special keys on your phone, such as the Dial, Hangup, volume, and softkeys 1 and 2. The screenshot I've included is of the whole MyMobileR application, but you can also use the application itself to take a screenshot just of the mobile device's screen.

MyMobileR can be set to zoom your mobile device's screen at 50%, 100%, 200% and 300%. I can't imagine why anyone would want to shrink it, but the 200% and 300% options may be nice for some folks. The only minor (looking a gift-horse in the mouth?) issues I found with the program are related to the viewing options; The borders of the regular view are sizeable, but you don't want to touch them. Changing the size does not alter the zoom, and you won't get scrollbars if you size it too small. To reset the borders to fit your mobile device screen, simply change the zoom level. I will note that the program handles the small, square screen of my Treo 700 just fine. Also, there is a 'skinned' view, which uses somewhat less space on the computer desktop. However, MyMobileR did not handle my Treo's screen well on that. But it may work just fine on a device with a more standard shape screen.

Overall, you can't beat MyMobileR's simplicity and price. Note that it only works on phones based on Windows Mobile 2003, 5 or 6. It will not work on a Blackberry, a Palm device running a Palm operating system, an iPhone, or any Linux or proprietary system.