Apache2 and Personal Web Browsing in Leopard

The answer to Friday's post is simple. Apple installed Apache 2, which has a different directory structure than that of apache.

The answer can be found on Apple's Forums. Also PHP is turned off by default, so be sure to edit the httpd.conf file to turn it back on.

From the post:
"I got PHP working with Leopard by modifying the httpd.conf file that you can get to by going to Go -> Go To Folder, /etc then going into the apache2 folder and copying httpd.conf to the Desktop (it won't let you edit in place). Find the line that says LoadModule php5_module etc...... and remove the # from the start. Save the file and drag it back into the apache2 folder, you'll have to authenticate to get it in there. Then restart apache by switching personal web sharing off and on in the sharing pref pane.

I had the same problem with my personal web sharing folder for my username not working on both the machines I installed it on (as an upgrade). The machines web sharing is working tho, just not the one for each user account. To fix it, create a file called shortusername.conf (where shortusername is your shortusername, eg, joebloggs and in it put this...

Directory "/Users/shortusername/Sites/"
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
/Directory

Save the file and put it into the apache2/users folder - restart personal web sharing and boom!, its working now."

I did all this in the terminal, sudo of course.

I hope this helps.

Apache post Leopard 10.5 install

The upgrade itself went fine. However, now I cannot see in a browser:
http://localhost/~ChrisCopeland

http://localhost works fine

I checked out 2-3 other leopard upgrades - it's consistent on all machines.

My (current) permissions are:
drwxrwxrwx+ 14 ChrisCopeland ChrisCopeland 476 Oct 26 14:20 Sites

Apple tech support thought I was on server. The dude was not aware that apache comes on all installs. Once we got over that hurdle he was still a little confused. I looked through the httpd.conf file but could not find anything out of the ordinary.

Can anyone else verify? Got any ideas?

Setup is:
Model Name: Mac Pro
Model Identifier: MacPro1,1
Processor Name: Dual-Core Intel Xeon
Processor Speed: 2.66 GHz
Number Of Processors: 2
Total Number Of Cores: 4
L2 Cache (per processor): 4 MB
Memory: 2 GB
OS X 10.5

A Complete Course in Electronics

usacomputertec over at YouTube has posted a 5 part series on how to use the Radio Shack, 28-280 Electronics Learning Lab.

I think I will pick this up to get a good refresher on my electronic circuits. It has been a long time since I picked up a soldering gun and went to town, so perhaps, if you are like me and need a refresher, you should consider one of these kits. The Radio Shack website has them for $65. I will drive down to the local RS store at lunch and see if they are instock.

Check out the video, it assumes you are a complete beginner, which if you are like me, you might as well start over from the start.

How To: The Urchin Data Extractor part 2

Well I said I would publish my script on Friday. Sorry about that, my wife and I moved into a new house over the weekend and I guess I was just a little over zealous.

Anyway. The script. In the past post I showed you how to (basically) setup the urchin script and make the command line calls to it. In this part, I will show you what I did to automate the process for ytd 2007, including all the months up to the current month. I utilized the bash quite a bit here. I'm sure there is a faster way to automate this without using as many tmp files, but I like to keep my data in stages.

Here is the script:
#!/bin/bash
cd /Users/ChrisCopeland/Apps/scripts/urchin
currentMonth=`date '+%m'`
currentdate=`date '+%Y%m%d'`
currentYear=`date '+%Y'`
i=1
let loopVar=currentMonth+1
echo "Monthly Reports Available :"

while [ $i -lt $loopVar ]; do
echo "$i"
let i=i+1
done
read -p "please select the month or enter ytd for year to date: " -e input
case "$input" in
'ytd')
perl u5data_extractor.pl --begin 20070101 --end $currentdate --max 21000 --report 1201 >>fileTMP;
;;
'1')
perl u5data_extractor.pl --begin 20070101 --end 20070130 --max 21000 --report 1201 >>fileTMP;
;;
'2')
perl u5data_extractor.pl --begin 20070201 --end 20070230 --max 21000 --report 1201 >>fileTMP;
;;
'3')
perl u5data_extractor.pl --begin 20070301 --end 20070330 --max 21000 --report 1201 >>fileTMP;
;;
'4')
perl u5data_extractor.pl --begin 20070401 --end 20070430 --max 21000 --report 1201 >>fileTMP;
;;
'5')
perl u5data_extractor.pl --begin 20070501 --end 20070530 --max 21000 --report 1201 >>fileTMP;
;;
'6')
perl u5data_extractor.pl --begin 20070601 --end 20070630 --max 21000 --report 1201 >>fileTMP;
;;
'7')
perl u5data_extractor.pl --begin 20070701 --end 20070730 --max 21000 --report 1201 >>fileTMP;
;;
'8')
perl u5data_extractor.pl --begin 20070801 --end 20070830 --max 21000 --report 1201 >>fileTMP;
;;
'9')
perl u5data_extractor.pl --begin 20070901 --end 20070930 --max 21000 --report 1201 >>fileTMP;
;;
'10')
perl u5data_extractor.pl --begin 20071001 --end 20071030 --max 21000 --report 1201 >>fileTMP;
;;
'11')
perl u5data_extractor.pl --begin 20071101 --end 20071130 --max 21000 --report 1201 >>fileTMP;
;;
'12')
perl u5data_extractor.pl --begin 20071201 --end 20071230 --max 21000 --report 1201 >>fileTMP;
;;
esac

#cleans the slash for easier editing
tr "/" "_" <>fileTMP2
#start of line removal
sed '
s/_index.cfm//
s/_&safe=vss//
s/_&adlt=strict//
s/.cfm//
s/,//
s/\ $//' <>fileTMP3
cat cleanThese | while read line; do
sed -ie "/$line/d" fileTMP3
done
less supplierTMP3 | cut -c 2-500 | grep -v "^ " | grep -v "^_" | grep -v "^#" >>fileTMPws
tr -s " " fileTMPcond
sed 's/ /,/' /Users/ChrisCopeland/Sites/urchinreports/file$currentYear-Month$input.csv
#cleans temp directory
rm fileTMP*
echo "your report is complete"

Now we can go through the script. The first part should be self explanatory, I am setting the values for the current date, month, and year (year is not yet implemented in my script, but will be soon). Then I ask the user for which report (monthly) they would like to generate, and show them a numerical list of the reports available (from 1 to current month). It will be in this case function that I will implement the current year, so that a user can get the month and year data needed. At the moment we are only interested in the year 2007.

Next comes the hairy part - the cleaning of useless data. I have a file for the most common items I want filtered. The file is called cleanThese (simple name). Before I open that file though, I want to clean certain characters and items which get skipped over do to the fact that this log file which is generated, will have a list of urls and paths in it. Paths and urls have weird characters sometimes, like ";" ":" "/" etc. Try passing these into a command line sometime and you will see how troublesome they can be. So let's get them out of there.

tr "/" "_" - replaced "/" with an underscore, which will make it easier to clean the rest of the log.

Now we throw this whole thing to sed - a great program.
sed '
s/_index.cfm//
s/_&safe=vss//
s/_&adlt=strict//
s/.cfm//
s/,//
s/\ $//' <>fileTMP3

we have a lot of oler cold fusion files, and some items that have a problem in the clean file, this sed command, which is a chained command, one per line, cleans these things out and leave nothing in the pattern's place. You can see now why I cleaned out the extra "/", I would have been passing a /// to sed, which it doesn't understand.

Now the cleanThese file:
cat cleanThese | while read line; do
sed -ie "/$line/d" fileTMP3
done
This reads each line of the cleanThese file (which I can modify at my desire) and replaces the entire line of the patten with nothing, effectively removing the line.

Then I want to clean up the formatting from the output of the cleaning:
less fileTMP3 | cut -c 2-500 | grep -v "^ " | grep -v "^_" | grep -v "^#" >>fileTMPws

This line will cut certain characters out (based on the original output), pass these to grep with an inverted search 3 times looking for different patterns, then writes that out to yet another tmp file.

The next line:
tr -s " " fileTMPcond
compresses all the space characters to one, outputs that to another file, which sed will take in, and replace the now single space, with a "," - effectively making this a csv file, which is names with the current month and set to a directory.

The next couple of lines clean the directory of tmp files and report to the user in the shell that the report is ready.

This will leave you a clean, importable, ready to query csv file just aching to be imported into a SQL engine of some sort.

Again if you want to run a different type of report, or use the case statement to generate a set of reports, you can visit my website to find a list of urchin reports available.

I want to find a good way to make this script available in a web interface at some point. I would also like to give the user a list of reports and years at the front of the application, just to help automate the process further.

Please Enjoy!

How To: The Urchin Data Extractor (u5data_extractor)

You can get the perl scripts for customizing Urchin data at the Google Urchin Support Page. I read the little documentation on this subject, which is a basic how to, without much resource. Urchin support firms charge something serious to get this kind of thing done, and here I am being a nice guy, giving away what I learned FOR FREE.

So let's begin with the lessons I learned.

1. Use some form of linux/unix. I could not, for the life of me, get any of these scripts to work with Windows and I think this is because of the path. The perl script is looking for a unix like path. I'm sure there are those people out there, smarter than I, who can get this to work on a windows server, but I am not one of them. The examples I give will be run from a Macintosh running OS X 10.4.10, ActiveState Perl, and the bash. In addition I would like to thank the wonderful folks (yet again) over at macosxhints forums as well as unix.com forums for helping me get my syntax correct in my scripts.

2. Use a step by step process.

3. Verify your data, and backup! The last thing you want to do is run an untested and "use at your own risk" script on your Urchin reports.

4. Do not always believe the available documentation.

5. When report testing, use small segments of data for your report. It saves time and you get to test your text scrubber faster.

Ok - now let's get to the logical process. What I wanted to do was to pull certain reports from Urchin and post them to a database, preferably some flavor of SQL.

The process will look something like this.
1. run perl script with start date, end date, report type, and number of items returned.
2. save report as a text file
3. scrub text file for bad characters, bad lines, and data which is not applicable.
4. comma delimit the file
5. hand csv file to sql import engine.

sounds easy right? It is for the most part.

The u5data_extractor script will do a lot of this work for you. This is the usage section of the script, which will also show up in the command line if you call the script with ~$ perl u5data_extractor. I removed the copyright and some other text for the purpose of posting to the blog.
###########################################################
# Usage: u5data_extractor.pl [--begin YYYYMMDD] [--end YYYYMMDD] [--help]
# [--language LA] [--max N] [--profile PROFILE]
# [--report RRRR] [--urchinpath PATH]
#
# Where:
# '--begin YYYYMMDD' specifies the starting date (default: one week ago)
# '--end YYYYMMDD' specifies the ending date (default: yesterday)
# '--help' displays this message
# '--language LA' specifies the language for the report. Available
# languages are: ch, en, fr, ge, it, ja, ko, po, sp, and sw
# '--max N' is the maximum number of entries printed in the top 10 report
# types (default is 10).
# '--profile PROFILE' specifies the profile to retrieve data from. The
# default is specified at the beginning of this script
# '--report RRRR is the 4-digit number for the report (default is 1102)
# Run this script with --help to see a list of available reports
# '--urchinpath PATH' specifies the path to the Urchin distribution.
# Note that you can edit the script and set your path as a default
###################################################

Giving the script your default path:
You will need to give the script the path to the Urchin Directory.
this is the line for my machine (following a unix path):
my $urchinpath = "/usr/local/urchin"; # Path to the Urchin distribution

Give the script your default profile:
You will need to give the script the default profile.
This is the line for a made up profile in the script.
my $profile = "My Default Profile"; # Name of the default profile
This is important - you do not have to use %20 to represent spaces if you are using the quotes. Urchin, by default, stores the profile directories with %20 for whitespace characters.

The report number is a difficult thing. Where do you find those reports? I found an article, somewhere, which shows the report numbers. Have no fear, I made a list for you of the urchin report numbers.

I will give an example, since none was really given for me. Let's say I want to run a report from Jan 01, 2007 to Jan 27, 2007 for the report "Visitors & Sessions"
so when you call the script, you will be using the following syntax:
perl u5data_extractor --begin 20070101 --end 20071027 --report 1903 --max 10

this will generate the output to the standard out (screen), which I will not post due to privacy reasons.

If you want to redirect the output feel free to do so
perl u5data_extractor --begin 20070101 --end 20071027 --report 1903 --max 10>>output.file

Tomorrow I will post my scrubbing process as well as the script I used to call backup the data and generate the reports.

Enjoy!

Using Bash Script to check for a server connection in OS X

Many thanks to robinwmills over at macosxhints forums for the assist on this. I needed a script which would check for a server connection, and if that server was not present, attach the server with a name and password.

The issue here is that smb connections from an OS X box to a Samba/Windows box often just quit. It is a known issue requiring you to jump through fiery hoops to re-establish the smb connection. All I needed to do was dump some csv files to a location and let SQL Server 2005 handle the rest.

Here is the code:
#!/bin/bash
#checks to see if share is mounted
stat /L/windows >& /dev/null
if ( $status ) then
echo 'setting up mount on /L'
osascript -e 'mount volume "smb://domain;username:password@10.0.0.0/share"'
else
echo 'alive'
fi
#file manupilation
umount /Volumes/share

From what robinwmills explains:
stat does soemthing like ls - it lists directory/file information. I can't remember why I used stat instead of ls, however I don't think there was any special reason.

Anyway, I have the Share "L" on /Volumes/L (and a sym link /L -> /Volumes/L)

stat /L/windows >& /dev/null

says "pipe stdout and errout from stat to no-where". stat sets $status to 0 for success. So I can test to see if there is a windows directory on L. If you don't use the /L symlink trick, you could do stat /Volumes/L/windows >& /dev/null

What I did was establish the connection (authenticating to the domain controller) for only the amount of time needed to copy the files, then disconnect by umount'ing the volume. I'm sure there was probably a more graceful way of handling the disconnect, but it gets the job done. I had not yet ever used applescript from the shell before, that turned out to be quite handy, as well as the line for connecting to the share with a domain, username, and password. I just passed this off to my local cron bad boy, and this is running like a charm. I have used shell in applescript - I will definitely have to read up more on this.

I found it difficult peering through the search engines for this type of solution, so I will make sure that it get the proper tags.

Enjoy!

Bye Bye XP

After upsetting me for the last time, I did something I never thought I would do...I took the plunge whole-heartedly into Linux.

Mind you this is not trying the flavor of the week on some secondary machine. I backed up all my non-os data, and wiped my XP load clean off my primary PC at home (it will have to do until I get another mac).

I tried SuSE Open Source 10.3, I have to say that although it was nice, I was still left with the impression that it was missing things, mainly dual monitor support. U will write more tomorrow about my SuSE 10.3 experience, good and bad.

So Ubuntu it is...dual monitors, FireFox, Perl, Python, and CrossOver/Wine (soon). I'm not really needing much more atm. I don't have to worry about licensing, OEM crap, or corruption (scripts in place back up my home folder to a thumb drive).

It has been a while, but it is nice to be back /127.0.0.1

Gateway refuses to downgrade Vista to XP

I tried to get a laptop downgraded from Vista Home to XP. This is the chat session from Gateway technical support. This was recorded at roughly 8pm CST on 10/8/07.
-----------------------------------
James_GWER7961 joined the incident.

James_GWER7961 says:
Thank you for contacting technical support. My name and badge number are James_GWER7961. Please hold while I review your issue.

James_GWER7961 says:
Hi Chris! How are you?

Chris Copeland says:
James I have had better - but thanks for asking

Chris Copeland says:
:)

Chris Copeland says:
I would like to know how to trade in my license of Vista for a license of XP

James_GWER7961 says:
According to your submission and my data, the serial number for your Gateway system is xxxxxxxxxx. Is this correct?

Chris Copeland says:
unless I'm reading it wrong - yes

James_GWER7961 says:
Thank you for verifying your information. I see that you would like to trade your operating system to Windows XP from Windows Vista.

Chris Copeland says:
yes please

James_GWER7961 says:
I apologize but we do not offer trade in for retail products and parts specially software.

Chris Copeland says:
so I have a license to an OS that I cannot use?

Chris Copeland says:
none of my applications are vista ready

James_GWER7961 says:
For us to further assist you with your concern, I need to have your complete address, purchase location of your computer and purchase date.

Chris Copeland says:
I will have to gather the information on the purchase location/date as I do not have the receipt with me, but my address is (removed for privacy) arlington tx

Chris Copeland says:
let me see if I can get the receipt.

Chris Copeland says:
I do not have the receipt, but I know it was the Best Buy in Mansfield Texas

James_GWER7961 says:
Please continue to hold while I register your system.

Chris Copeland says:
Thank you

James_GWER7961 says:
Thank you for holding. I have successfully registered your system under your account.

Chris Copeland says:
thank you

James_GWER7961 says:
My information shows that your computer came with Windows Vista operating system preinstalled.

James_GWER7961 says:
Is this correct?

Chris Copeland says:
unfortunately yes

James_GWER7961 says:
Well, the software that also came with it are Windows Vista compatible.

Chris Copeland says:
yes that is true, but the software I own outside of the bundle does not.

James_GWER7961 says:
So its a third party software?

Chris Copeland says:
yes - and it is not working with Vista

James_GWER7961 says:
Have you tried to contact the manufacturer of the software that you installed regarding this?

Chris Copeland says:
yes. I was instructed to revert to XP SP2

James_GWER7961 says:
You would need to purchase a retail version of Windows XP, Chris. We are unable to replace the current operating system installed to yoru computer which is Windows Vista.

Chris Copeland says:
thank you

Chris Copeland says:
I assume I can keep a copy of this chat report

James_GWER7961 says:
To save your chat log, please click on the "View Live Log" button at the bottom left side of the chat window. It will open a chat log window. Please use the option to "Save log to disk". Keep in mind you do not need a disk or cd. You can simply save the log to your system.

James_GWER7961 says:
I hope that I was somehow able to provide you the information that you need.
----------------------------------

Tried as I did, I was told that I must go buy a copy of XP, retail. So now I have a copy of Vista that I am never going to use, and no way to get rid of it. I simply do not accept the terms of the Vista license Agreement. According to the EULA on Microsoft's site, the downgrade only applies to Vista Business and Ultimate users, not Home Edition (which I have pre-installed). So what do I do about a refund then? I simply do not want Vista loaded, but it looks like I am either being forced to run Vista, or being forced to buy a copy of XP. I want to make it clear that there is currently no operating system on the laptop. The Vista license is not installed on any machine.

PDF in iTunes

The idea of having PDF for my research was certainly not a new idea. I turned several students on to the idea in grad school. I even turned in the CD as my compilation of my research works which was a lot simpler than turning in a few boxes of research paper.

I read this fine article on Make this morning:
Digg user enjayenel wrote "I love using iTunes as a PDF library tool. I have hundreds of PDF manuals that I need organized access to. I just add them into the iTunes Library, edit the ID3 tags, create a smart playlist to group them together, and turn on the Browse feature (command-b) to get quick navigation to the PDF I am looking for based on title, product name (artist tag), and version (album tag). Double clicking the PDF opens it in your default PDF viewer".

Holy crap batman. I can't wait to get home to my several thousand pdf files and use one of the best applications yet. This functionality will change how I do research from now on. I have now read up on (almost) everyone's opinion regarding pdf files in iTunes, and there are some bit I like and I don't like. I think the idea of turning off the "copy to iTunes directory" is a bad thing. Let me tell you why...one word...backups. If you let your iPod auto sync with iTunes, you will automatically have a backup of your work, no more late night crashes, even more so, your information is organized.

To get that information organized we will have to modify the id3 tags in the meta data of the file in iTunes. I also have to agree that PDF support should be a plugin from apple. On the mac, imho, nothing beats preview and it's abilities, Foxit Reader may do it for me, I will test it tonight.

I am now wondering if there is an automated way to import the data, perhaps the title information from the pdf itself. I know this is all just stored in the XML, but there must be a way. I will have to check the automator script archives around the web to see if there is anything useful. Just the use of smart playlists now have me jumping for joy.

TuneTagger

I have been looking for an application which could easily attach lyrics to my mp3 and aac file in iTunes. I read about TuneTagger yesterday, tried it and liked it right off the bat (which is unusual for me).

I like the way TuneTagger checks the tags against the CDDB then displays what is called the "approval pane" which lets you chose which information to embed into the song file.

This even works with non-syncing iPods (at least it works with 5th gen). My iPod is not set to sync, I do it all manually, as long as iTunes plays a song, TuneTagger will verify the tags in the current or previous song.

This is a great app for $17.