Comments on: MaxMind GeoIP: Import MaxMind City CSVs into SQL Server 2005 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/ ls /usr/lolcat Wed, 20 Aug 2008 02:12:53 +0000 http://wordpress.org/?v=2.3.1 By: Charles http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-10301 Charles Thu, 31 Jul 2008 15:16:39 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-10301 As solution to the problem I wrote about above, you're look logic can be done like this: SELECT TOP 1 case when @intIPNum > [Last] then null else locid end FROM GeoIPCityBlocks WHERE @intIPNum >= startIpNum ORDER BY startIpNum DESC This query will return NULL in the case @intIPNum is not found between any ranges within the database. Charles As solution to the problem I wrote about above, you're look logic can be done like this:

SELECT TOP 1
case when @intIPNum > [Last] then null else locid end
FROM GeoIPCityBlocks WHERE @intIPNum >= startIpNum ORDER BY startIpNum DESC

This query will return NULL in the case @intIPNum is not found between any ranges within the database.

Charles

]]> By: Charles http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-10298 Charles Mon, 28 Jul 2008 16:31:56 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-10298 Your lookup logic incorrectly assumes that endIpNum at row X is always one less than startIpNum at row X+1. That is, your code is assuming contiguous IP ranges like this: startIpNum endIpNum locId 67276848 (4.2.144.48) 67277023 (4.2.144.223) 223 67277024 (4.2.144.224) 67277031 (4.2.144.231) 994 67277032 (4.2.144.232) 67277039 (4.2.144.239) 223 67277040 (4.2.144.240) 67277047 (4.2.144.247) 33539 67277048 (4.2.144.248) 67277055 (4.2.144.255) 4027 67277056 (4.2.145.0) 67277215 (4.2.145.159) 15886 But this is not true for all rows. Here is some data directly from the database ordered by startIpNum: startIpNum endIpNum locId 368674304 (21.249.134.0) 385875967 (22.255.255.255) 223 386665696 (23.12.12.224) 386665727 (23.12.12.255) 58390 386666240 (23.12.15.0) 386666367 (23.12.15.127) 58393 393849728 (23.121.171.128) 393849735 (23.121.171.135) 32371 402653184 (24.0.0.0) 402653695 (24.0.1.255) 118769 402653696 (24.0.2.0) 402653951 (24.0.2.255) 57771 Your logic returns a locId of 58393 for all the IP address between 23.12.15.128 and 23.121.171.127, but in actuality those addresses have no entry in the database. That's 7,183,361 IP addresses that get resolved to Great Britain when in fact the database doesn't say that. Please correct me if I'm wrong. Charles Your lookup logic incorrectly assumes that endIpNum at row X is always one less than startIpNum at row X+1. That is, your code is assuming contiguous IP ranges like this:

startIpNum endIpNum locId
67276848 (4.2.144.48) 67277023 (4.2.144.223) 223
67277024 (4.2.144.224) 67277031 (4.2.144.231) 994
67277032 (4.2.144.232) 67277039 (4.2.144.239) 223
67277040 (4.2.144.240) 67277047 (4.2.144.247) 33539
67277048 (4.2.144.248) 67277055 (4.2.144.255) 4027
67277056 (4.2.145.0) 67277215 (4.2.145.159) 15886

But this is not true for all rows. Here is some data directly from the database ordered by startIpNum:

startIpNum endIpNum locId
368674304 (21.249.134.0) 385875967 (22.255.255.255) 223
386665696 (23.12.12.224) 386665727 (23.12.12.255) 58390
386666240 (23.12.15.0) 386666367 (23.12.15.127) 58393
393849728 (23.121.171.128) 393849735 (23.121.171.135) 32371
402653184 (24.0.0.0) 402653695 (24.0.1.255) 118769
402653696 (24.0.2.0) 402653951 (24.0.2.255) 57771

Your logic returns a locId of 58393 for all the IP address between 23.12.15.128 and 23.121.171.127, but in actuality those addresses have no entry in the database. That's 7,183,361 IP addresses that get resolved to Great Britain when in fact the database doesn't say that.

Please correct me if I'm wrong.

Charles

]]> By: Tim http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-1214 Tim Wed, 09 Jan 2008 13:48:33 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-1214 You can use another technique proposed by IP2Location. It is using format file to define CSV format for command line import. http://www.ip2location.com/faqs-ip-country.aspx#9 You can use another technique proposed by IP2Location. It is using format file to define CSV format for command line import.

http://www.ip2location.com/faqs-ip-country.aspx#9

]]> By: Emil Zegers http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-881 Emil Zegers Wed, 07 Nov 2007 10:38:33 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-881 I've used your scripts to import the MaxMind data in MSDE (Microsoft SQL Server Desktop Engine 8.00.760). To get this done, the following adjustments to the scripts had to be made: SQL script: - Changed 'sys.databases' and 'sys.objects' to 'sysdatabases' and 'sysobjects' - Changed 'WHERE object_id = OBJECT_ID' to 'WHERE id = OBJECT_ID' - Added index name to INDEXDEFRAG: DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCountry,IDX_countryStarIPtNum) DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCountry,IDX_CountryCode) DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCityBlocks,IDX_blocksStartIpNum) DBCC INDEXDEFRAG (maxmindgeoip,geoIPCityLocation,IDX_LocationID) - Changed 'GRANT CONNECT TO [guest]' to 'EXECUTE sp_grantdbaccess [guest]' - Added percentage and truncate parameter to SHRINKDATABASE DBCC SHRINKDATABASE (maxmindGeoIP,10,NOTRUNCATE) Thanks for your scripts! Regards, Emil I've used your scripts to import the MaxMind data in MSDE (Microsoft SQL Server Desktop Engine 8.00.760).

To get this done, the following adjustments to the scripts had to be made:

SQL script:

- Changed 'sys.databases' and 'sys.objects' to 'sysdatabases' and 'sysobjects'

- Changed 'WHERE object_id = OBJECT_ID' to 'WHERE id = OBJECT_ID'

- Added index name to INDEXDEFRAG:

DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCountry,IDX_countryStarIPtNum)
DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCountry,IDX_CountryCode)
DBCC INDEXDEFRAG (maxmindgeoip,GeoIPCityBlocks,IDX_blocksStartIpNum)
DBCC INDEXDEFRAG (maxmindgeoip,geoIPCityLocation,IDX_LocationID)

- Changed 'GRANT CONNECT TO [guest]' to 'EXECUTE sp_grantdbaccess [guest]'

- Added percentage and truncate parameter to SHRINKDATABASE

DBCC SHRINKDATABASE (maxmindGeoIP,10,NOTRUNCATE)

Thanks for your scripts!

Regards,

Emil

]]> By: James Hooker http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-624 James Hooker Wed, 25 Apr 2007 01:47:00 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-624 Fantastic, thank you very much! This little package does exactly what it says on the tin! :) Fantastic, thank you very much! This little package does exactly what it says on the tin! :)

]]> By: Drew Freyling http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-623 Drew Freyling Fri, 02 Mar 2007 07:07:44 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-623 I'm guessing the overhead of setting that everytime the proc is run is greater than the performance gain from not row/table locking on each read. I'm guessing the overhead of setting that everytime the proc is run is greater than the performance gain from not row/table locking on each read.

]]> By: Chrissy http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-622 Chrissy Thu, 01 Mar 2007 22:58:42 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-622 Hey Drew, That seemed very possible so I tried it out by adding the following to the store procedure: SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED Strangely, it actually slowed down the query. The average resolution for 255 IPs without the explicit transaction level was 189ms on average and with the transaction level set, it was 204 ms. Hey Drew,
That seemed very possible so I tried it out by adding the following to the store procedure:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

Strangely, it actually slowed down the query. The average resolution for 255 IPs without the explicit transaction level was 189ms on average and with the transaction level set, it was 204 ms.

]]> By: Drew Freyling http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-621 Drew Freyling Wed, 28 Feb 2007 02:29:46 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-621 Will setting the transaction isolation level to read uncommitted( thereby removing the locking overhead) make the IPtoLocation proc perform any quicker? Will setting the transaction isolation level to read uncommitted( thereby removing the locking overhead) make the IPtoLocation proc perform any quicker?

]]> By: jrp http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-620 jrp Sat, 24 Feb 2007 13:33:49 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-620 The answer to the permissions problem seems to be to right click on unzip.exe and click Unblock The answer to the permissions problem seems to be to right click on unzip.exe and click Unblock

]]> By: jrp http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-619 jrp Fri, 23 Feb 2007 23:04:40 +0000 http://blog.netnerds.net/2007/01/maxmind-geoip-import-maxmind-city-csv-into-sql-server-2005/#comment-619 I still get "permission denied" problems running the vbs script. Suggestions for debugging? I still get "permission denied" problems running the vbs script. Suggestions for debugging?

]]>