VBScript & SQL: Programatically Find the Location of an IP Address

Filed under: Networking, Quick Code, SQL Server, VBScript — Written by Chrissy on Tuesday, January 30th, 2007 @ 7:21 pm

This is more of a post for proper keywords. My previous entry titled "Import MaxMind City CSVs into SQL Server 2005" assumes the user has already conducted research to find that MaxMind's free GeoIP database provides the functionality to look up the location of an IP address. So, if you wish to quickly and easily find an IP address' location, do the following.

1. Ensure you have SQL Server installed somewhere. If you don't currently have it installed,
    you can download SQL Server 2005 Express for free.
2. Read this post about importing the necessary (and free) database for IP lookups.
    a. Download zip & unzip
    b. Read README.txt
    c. Copy the unzip.exe to your Windows directory
    d. Run the included SQL File
    e. Run the included VBS file
3. Create a VBS file to access the information.
4. Double-click the VBS file and enter the IP address

LookupIPAddress.vbs

  strSQLServer = "localhost"
  strDBName = "maxmindGeoIP"
 
  strIPaddr = InputBox("Enter the IP address:","IP To Location")
 
  If Len(strIPaddr) = 0 Then Wscript.Quit
 
  Set objRS = CreateObject("ADODB.recordset")
  strConnstring =  "Driver={SQL Server};Server=" & strSQLServer & ";Database=" & strDBName & ";Trusted_Connection=Yes;"
  strSQL = "EXEC usp_IPtoLocation '" & strIPaddr & "'"
  objRS.Open strSQL, strConnstring, 1, 1
 
  If objRS.eof And objRS.bof Then
    MsgBox "Something is ultra-broken. Is your database populated? " 'Everything should return at least something.
  Else
    strLocation = objRS("location")
      If InStr(strLocation,"Unknown") > 0 Then
      MsgBox "The IP address, " & strIPaddr & ", cannot be found in the database :(",64,"IP To Location"
      Else
      MsgBox  strIPaddr & " resolves to.." & vbCrLf & vbCrLf & objRS("location"),64,"IP To Location"
      End If
   End If
 
   Set objRS = Nothing

Then, finding the location is as easy as double-clicking the file, entering an IP, and pressing "OK"....

In less than 1 millisecond, (after the initial stored procedure execution), your results will show up

T-SQL: Quickly Kick All Users Out of a SQL Server Database

Filed under: Quick Code, SQL Server — Written by Chrissy on Tuesday, January 30th, 2007 @ 3:13 pm

There are a few ways to do this but here's one that I like. I believe it works in SQL Server 7, 2000 and 2005.

ALTER DATABASE myDatabaseName SET SINGLE_USER WITH ROLLBACK IMMEDIATE
GO
USE myDatabaseName
GO

Go ahead and do what you need to do then set it back:

ALTER DATABASE myDatabaseName SET MULTI_USER

BCP: Remove Quotes from CSV Import using BCP Format File

Filed under: SQL Server — Written by Chrissy on Monday, January 29th, 2007 @ 8:53 pm

I'd like to start this post by saying that BCP is limited and ghe-toe, but it gets the job done to some degree. One of the guys over at SQLteam.com wrote a really informative post on Flat File Bulk Import methods speed comparison in SQL Server 2005 that you should check out if you are into that sort of thing. BCP ranks last on the list but I recently decided to use it anyway for an informal import project I worked on.

So what I found was that BCP doesn't play well with INTs/BIGINTs and that it's best to just use "SQLCHAR" and let SQL Server do it's implicit conversions. I also learned that it's a pain to remove quotes but I managed to do so with some hackery and want to post it here as I've seen others needing the same thing in newsgroups. Here are two different samples that I've worked with: the bolded fields are from a sample CSV file and the blue boxes are the corresponding bcp.fmt files.

locId,country,region,city,postalCode,latitude,longitude,dmaCode,areaCode
68954,"US","LA","Kaplan","70548",29.9845,-92.3224,642,337

7.0
9
1       SQLCHAR    0       255       ",""    1     locId
2       SQLCHAR    0       255       "",""    2     country
3       SQLCHAR    0       255       "",""    3     region
4       SQLCHAR    0       255       "",""    4     city
5       SQLCHAR    0       255       "","     5     postalCode
6       SQLCHAR    0       255       ","    6     latitude
7       SQLCHAR    0       255       ","    7     longitude
8       SQLCHAR    0       255       ","    8     dmaCode
9       SQLCHAR    0       255       "\n"   9     areaCode

startIpNum,endIpNum,locId
"33996344","33996351","16552"

7.0
5
1       SQLCHAR    0       255       """     0     firstQuote
2       SQLCHAR    0       255       "",""     1     startIpNum
3       SQLCHAR    0       255       "",""     2     endIpNum
4       SQLCHAR    0       255       """    3     locId
5       SQLCHAR    0       255       "\n"    0     lastQuote

Then run: bcp dbName.dbo.tableName in mydata.csv -F 2 -S sqlSrvr -T -f bcp.fmt
(-F tells bcp which line to start on, -T is trusted connection, -S is the SQL Server name)

If you are wondering what all the fields are for, this is a really helpful diagram (as found in SQL 7 Books Online)

Again, SQLBIGINT never worked for me under SQL Server 2005 (even when I set the format file version to 9.0) so I'd just recommend using SQLCHAR (varchar equiv) and let SQL Server do the implicit conversion for you.

MaxMind GeoIP: Import MaxMind City CSVs into SQL Server 2005

Filed under: Networking, Quick Code, SQL Server — Written by Chrissy on Monday, January 29th, 2007 @ 5:45 pm

Inspired by kitty0's GeoIP Demo, I created two scripts which automatically import the free MaxMind GeoIP database (which resolves IP addresses to city locations) into SQL Server using VBS and BCP. The VBS file can be set as a Scheduled Task for MaxMind's free once-a-month updates. I also created a SQL Server UDF to access the data. Resolving the location of an IP is then as easy as running the following query:

SELECT dbo.IPtoLocation('208.65.15.0') as ipLocation (demo)

UDFs, like stored procedures, are pre-compiled and single results, after the intital compilation/first run, are returned in as little as 0.xx milliseconds. I've seen this script process 3000 hostnames a second on a dual proc Macbook (running Windows 2003/SQL 2005) with no load! A huge thanks goes out to Lee Holmes for helping with that. I was so focused on the indexes that I didn't pay attention to the actual data. Once Lee pointed this out, I was able to create a query which query a max of 3000 distinct recs/second as opposed to 6 recs/sec!

Without further ado, you can download the zip of importMaxmindGeoIP files here. This zip includes the following files:
1. importMaxmindGeoIP.sql
2. importMaxmindGeoIP.vbs
3. GNU unzip.exe (and it's license)
4. README.txt

The SQL file creates a database called maxmindGeoIP. It then creates 3 tables, 3 sample UDFs and 3 stored procedures. The database has been indexed as needed and queries have been optimized a ton.

importMaxMindGeoIP.sql

/*
Created by Chrissy LeMaire (clemaire@gmail.com)
Website: http://netnerds.net/
 
! - Run this prior to running importMaxmindGeoIP.vbs - !
 
This script
1. Creates a database named maxmindGeoIP. IF YOU ALREADY HAVE A DB NAMED MAXMINDGEOIP, THIS SCRIPT WILL DROP IT.
2. Creates a 'bcp prep' stored procedure which creates the necessary tables
    - This stored proc is run each time the importMaxmindGeoIP.vbs file is run
3. Creates another bcp related stored procedure which creates the necessary indexes, defrags the indexes and
   shrinks the database. The database is usually about the same size as the CSV (~55MB-65MB as of Jan07)
    - This stored proc is run each time the importMaxmindGeoIP.vbs file is run
4. Creates a few sample stored procedures and user-defined functions
5. Allows guest accounts to connect to and read the db.
6. The bottom of this file contains example usage such as
    -Select '208.65.15.0' as IP,maxmindGeoIP.dbo.IPtoLocation('204.65.15.0') as ipLocation
7. Remember that SQL UDFs and Stored Procedures are pre-compiled so the results will take a little longer the
   first time the function or procedure is run.
 
NO WARRANTIES, USE THIS AT YOUR OWN RISK, etc.
*/
 
On second thought, please just download the rest from the link above (the SQL file or the zip). It's quite long.

The VBS checks the MaxMind site for some zips, downloads them to %TEMP%, unzips to %TEMP%, creates the necessary BCP format files, imports the data and deletes all unnecessary files. It also calls two procedures which prep the database for an import and then cleans up the data and indexes it. This import takes 2-10 minutes to run, depending on a whole lot of variables. Please read the comments section to see the requirements (such as BCP and the login/pass for BCP).

importMaxmindGeoIP.vbs

'****************************************************************************
' Created by Chrissy LeMaire (clemaire@gmail.com)
' Website: http://netnerds.net/
'
' This script downloads the GeoLite CSV files from Maxmind.com
' and uses bcp to transform it into SQL Server data. Run this once
' on the second day of each month.
'
' ! - Run importMaxmindGeoIP.sql prior to running this script - !
' ! - Follow the instructions in README.txt or this won't work - !
'
' 1. You must download "unzip.exe" either from my server at
'     http://blog.netnerds.net/code/importMaxmindGeoIP.zip
'     http://sourceforge.net/projects/gnuwin32 and save it in your
'     PATH (ie. C:\windows)
'
' 2. You must execute the SQL statements found in the zip
'     http://blog.netnerds.net/code/importMaxmindGeoIP.zip prior to running this
'     script
'
' 3. The SQL Script creates a new database called MaxMindGeoIP
'     as well as some stored procs and UDFs. If you change anything
'     in the SQL script, make sure you update the BCP statements
'     below
'
' 4. Run this script on a machine with BCP and a trusted connection to the
'     SQL Server. If you don't have a trusted connection (a domain), change
'     the BCP parameters below (remove -T and add -u,-p).
'
' 5. Run this script at least once from the console. You may be prompted to
'     answer yes to the question "Do you wish to run this program?"
'     for both the vbs file and unzip.exe.  Uncheck the box
'     "Always ask before opening this file" and click "Run"
'
' I have only tested this on SQL Server 2005 and Windows Server 2003
'
' "What it does"
' 1. Downloads the GeoLite CSVs from Maxmind.com to the %TEMP% directory
' 2. Unzips the files into the same directory
' 3. Creates the necessary BCP format files
' 4. Executes bcp to bulk import the file
' 5. Executes a stored procedure which adds indexes to the tables and shrinks the DB.
' 6. Deletes all downloaded/extracted/created files once it's done with them
'
' NO WARRANTIES, USE THIS AT YOUR OWN RISK, etc.
'*****************************************************************************
 
'Meh, please just download this from the link above.

Once the tables are populated, you can do just about anything you'd like with the information. If you'd like to see it in action, click here for a demo on your IP address. The data is capable of returning longitude and latitude so you can also plug this in with Google Maps data (as seen here).

If you'd like this capability for your SQL Server database, download importMaxmindGeoIP.zip, read the README.txt and have fun. Just as an FYI, if you keep track of the IPs that hit your website, here's some sample code

SELECT ipaddr,maxmindgeoip.dbo.IPtoLocation(ipaddr)
FROM hitcounter WHERE dateadded between DATEADD(hh,-24,GETDATE()) and GETDATE()
GROUP BY ipaddr

The GROUP BY basically makes the ipaddr DISTINCT so this shows you where your visitors have come from for the past 24 hours. See the demo which resolves the locations for my last 25 visitors in real time. I also used the script below to query 3000 records a second.

SELECT dbo.iptolocation('24.0.175.222') as ipLocation -- ensure the function runs at least once to compile
GO
SELECT DISTINCT TOP 3000 ipaddr INTO ipaddrs FROM hitcounter -- populate and index a table with 3000 distinct IPs. of course, you need a table with at least 3000 distinct IPs.
GO
CREATE CLUSTERED INDEX IDX_ipaddrs ON dbo.ipaddrs (ipaddr ASC)
GO
DECLARE @startTime datetime
SET @startTime = getDate()
SELECT dbo.iptolocation(ipaddr) from ipaddrs
PRINT 'Your query took '+CAST(DATEDIFF(ms,@startTime,GETDATE()) AS varchar)+' milliseconds to run.'

I can consistently get 1063 and 1050 millisecond runtimes :-D
UPDATE: The original VBS file was set to download the first zip it saw in "http://www.maxmind.com/download/geoip/database/GeoLiteCity_CSV/". I didn't expect Maxmind to keep it archives there but now the updated script looks for the last zip (and presumably the most updated), not the first.

VI: Automatically Set vi To Support Windows-Compatible Carriage Returns

Filed under: Linux, Tech Stuff — Written by Chrissy on Friday, January 26th, 2007 @ 11:03 pm

Earlier today, I pasted some code into a Linux-based vi via putty/ssh, saved it, zipped it, and downloaded it to a Windows machine. The result was pooh; the code ran but was garbled in notepad. I wondered if I could somehow run a command in Linux to clean/convert all the carriage returns. One of my friends at Microsoft, Lee, suggested unix2dos (and its nemesis dos2unix).. just what I was looking for! However, if I didn't want to run unix2dos each time I saved the file, I could set vi's file format to dos at startup by adding the following alias to my .bashrc file

alias vi='vi -c "set ff=dos"'

Not such a great idea for unix-centric folk but a great thing for me :) As for a Windows equivalent of unix2dos... Lee, a developer on the Powershell team, just told me that you can run the following in Powershell to accomplish the conversion: Get-Content unixfile.txt | Set-Content dosfile.txt. I really need to get around to scripting in Powershell; it will no doubt save me a ton of time in the long-run. Until then, though, I'll continue to cater to old school VBScript-loving Windows Admins.

T-SQL UDF: Convert Traditional IP into Numeric IP

Filed under: Networking, Quick Code, SQL Server — Written by Chrissy on Thursday, January 25th, 2007 @ 12:46 pm

This user-defined function (only available in SQL 2000 and above) is part of a larger project I'm working on. PARSENAME in SQL Server is my new favorite function.. Thanks to Jeff Smith for the heads up.

CREATE      FUNCTION [dbo].[IPtoNumeric]  (@strIP varchar(255))
RETURNS bigint
AS
BEGIN
DECLARE @intIPNum bigint
        IF (LEN(@strIP)-LEN(REPLACE(@strIP,'.','')))/LEN('.') = 3   -- check to ensure there are 3 dots
    SET @intIPNum = (16777216 *  CAST(PARSENAME(@strIP,4) as bigint) + 65536 * PARSENAME(@strIP,3) + 256 * PARSENAME(@strIP,2) + PARSENAME(@strIP,1))
RETURN @intIPNum
END

You can quickly test this by issuing the following query:
SELECT dbo.IPtoNumeric('24.0.175.222') as numericIP

VBScript: Download and Save a Binary File

Filed under: Quick Code, VBScript — Written by Chrissy on Wednesday, January 24th, 2007 @ 12:03 pm

Update: If you are an iTunes user that needs VBScript, this page won't help you. Please visit the following two sites for two different solutions: Apple Docs and KeathMilligan.net.

This is an old script I dug up which I call "fileFetch". It's a script that downloads a binary file using XMLHTTP and saves it using an ADO stream. This is handy for nightly downloads and will work within SQL Server DTS packages.

Quick Code

' Set your settings
    strFileURL = "http://www.domain.com/file.zip"
    strHDLocation = "D:\file.zip"
 
' Fetch the file
    Set objXMLHTTP = CreateObject("MSXML2.XMLHTTP")
 
    objXMLHTTP.open "GET", strFileURL, false
    objXMLHTTP.send()
 
    If objXMLHTTP.Status = 200 Then
      Set objADOStream = CreateObject("ADODB.Stream")
      objADOStream.Open
      objADOStream.Type = 1 'adTypeBinary
 
      objADOStream.Write objXMLHTTP.ResponseBody
      objADOStream.Position = 0    'Set the stream position to the start
 
      Set objFSO = Createobject("Scripting.FileSystemObject")
        If objFSO.Fileexists(strHDLocation) Then objFSO.DeleteFile strHDLocation
      Set objFSO = Nothing
 
      objADOStream.SaveToFile strHDLocation
      objADOStream.Close
      Set objADOStream = Nothing
    End if
 
    Set objXMLHTTP = Nothing

The New Look of Google Groups

Filed under: Tech Stuff — Written by Chrissy on Wednesday, January 24th, 2007 @ 11:26 am

Wow, have ya'll seen the new Google Groups interface? It's sex on legs.

Now if only they'd put back "Groups" in the main Google navigation text.

ASP/VBScript: Send E-Mail with CDO

Filed under: Quick Code, VBScript — Written by Chrissy on Monday, January 22nd, 2007 @ 4:15 am

I've used CDONTS for years, even though it hasn't been included in a Windows Server release since NT4. Anytime I needed to send mail, I'd copy over CDONTS.dll, register it then use my old CDONTS code. Not sure why I resisted it for so long.. I think the weird Configuration Fields were too odd to accept. But I've finally accepted them.. about 3 good years after ASP went out of style. Here's a less-than-timely code snippet:

This is the sound of settling.

    Set objCDO = CreateObject("CDO.Message")
    objCDO.Subject = "Update from Web App."
    objCDO.From = "Web App <webapps@me.com>"
    objCDO.To = "yomomma@gmail.com"
    objCDO.TextBody = "Hello," & vbCrLf & vbCrLf & "A new order has been placed by " & userName & "."
    objCDO.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 'cdoSendUsingPort (1 = local, 3 = Exchange)
    objCDO.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "mail.me.com"
    objCDO.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
    objCDO.Configuration.Fields.Update
    objCDO.Send
    set objCDO = Nothing

Classic ASP: "Push" File Downloads from A Directory Outside the Application Root

Filed under: IIS, Quick Code, VBScript — Written by Chrissy on Monday, January 22nd, 2007 @ 4:04 am

This is some super old code but I used it recently and figured I'd archive it on this site for my future reference. The sample code below aims to allow authenticated users to download files which are not available via direct download (ie. files within the web root). The script accomplishes this by doing the following:

1. Checks to see if the user is logged in (your method may vary)
2. Sets the root directory location
3. Checks to see if the file exists, if so...
4. Retrieves the filesize and adds the appropriate HTTP headers including content disposition, filename, content type and filesize.
5. Uses a binary stream to "push" the download

Save this file as download.asp and call it with the filename in the querystring. Example: http://domain.com/downloads/download.asp?filename=myfile.pdf. Also, be sure to give read permissions to IUSR_SvrName to the root directory. Change the authentication requirements as needed:

download.asp

<%
If session("loggedIn") = True Then
 
  strFilePath = "D:\webfiles\downloads"  & request.querystring("filename")
 
  Set objFSO = Server.CreateObject("Scripting.FileSystemObject")
    If objFSO.FileExists(strFilePath) Then
      Set objFile = objFSO.GetFile(strFilePath)
      intFileSize = objFile.Size
      Set objFile = Nothing
 
      strFileName = request.querystring("filename")
      strFileName = replace(request.querystring("filename")," ","-")
      Response.AddHeader "Content-Disposition","attachment; filename=" & strFileName
 
      Response.ContentType = "application/x-msdownload"
      Response.AddHeader "Content-Length", intFileSize
 
      Set objStream = Server.CreateObject("ADODB.Stream")
        objStream.Open
        objStream.Type = 1 'adTypeBinary
        objStream.LoadFromFile strFilePath
        Do While Not objStream.EOS And Response.IsClientConnected
        Response.BinaryWrite objStream.Read(1024)
        Response.Flush()
        Loop
        objStream.Close
      Set objStream = Nothing
    Else
      Response.write "Error finding file."
    End if
  Set objFSO = Nothing
End If
%>

Note: Even if the file is a PDF and a third-party application such as Adobe Reader is set to open the file within the browser, the code below will override that and force a download (by using the "application/x-msdownload" content type).

[The UPDATE below is no longer accurate as an alternative solution has been given below in the comments and subsequently, was added to the code (Thanks a bunch, David!). I wanted to leave it for Googlers looking for a solution, however]

UPDATE: Someone wrote to let me know that they were encountering the error "Response Buffer Limit Exceeded". As it turns out, IIS 6's ASPBufferingLimit is set to a measly 4MB (4194304 bits) so any file over 4MB would produce this error. To fix this issue, you will have to have access to IIS either via the command line or the MMC. Here's how to change the buffering limit via the command line:

************ NOTE: An easier solution is to use the updated Do While/Flush procedure given in the code *****************

cd C:\inetpub\adminscripts
cscript adsutil.vbs set /w3svc/aspbufferinglimit 4294967295

That's a buffering limit of more than 4 Gigabytes. Personally, I'd lop off the last digit and make that number closer to 430MB. Running that script worked immediately on my test machine, even though I do not have "Enable Direct Metabase Edit" checked in IIS' Properties. If it doesn't work for you, restart IIS and see if it works.