T-SQL: Parse Top Level Domain from URL

Filed under: Quick Code, SQL Server — Written by Chrissy on Wednesday, January 17th, 2007 @ 1:12 pm

I keep track of all hits to my website RealCajunRecipes.com in a SQL table called hitcounter which has columns for the user's IP, browser, referring URL and the date. Recently, I saw a surge in traffic and wanted to know which domains were sending the traffic our way. After getting tired of issuing ad-hoc queries that included WHERE clauses like "where referer like '%google%'", I created a SQL Server user-defined function (UDF) to extract the domain from the referring URL.

parseurl.sql

CREATE      FUNCTION [dbo].[parseURL]  (@strURL varchar(1000))
RETURNS varchar(1000)
AS
BEGIN
  IF CHARINDEX('http://',@strURL) > 0 OR CHARINDEX('https://',@strURL) > 0
        -- Ghetto-tastic
        SELECT @strURL = REPLACE(@strURL,'https://','')
        SELECT @strURL = REPLACE(@strURL,'http://','')
        SELECT @strURL = REPLACE(@strURL,'www','')
        -- Remove everything after "/" if one exists
        IF CHARINDEX('/',@strURL) > 0 (SELECT @strURL = LEFT(@strURL,CHARINDEX('/',@strURL)-1))
 
        -- Optional: Remove subdomains but differentiate between www.google.com and www.google.com.au
        IF (LEN(@strURL)-LEN(REPLACE(@strURL,'.','')))/LEN('.') < 3 -- if there are less than 3 periods
          SELECT @strURL = PARSENAME(@strURL,2) + '.' + PARSENAME(@strURL,1)
        ELSE -- It's likely a google.co.uk, or google.com.au
          SELECT @strURL = PARSENAME(@strURL,3) + '.' + PARSENAME(@strURL,2) + '.' + PARSENAME(@strURL,1)
RETURN @strURL
END

This script does the following:
1. Checks to see if the string is an URL
    (example: str = http://www.search.google.com.au/?q=netnerds)
2. Removes http, https and www (str = search.google.com.au/?q=netnerds)
3. Removes everything after the slash (str = search.google.com.au)
4. Removes excessive subdomains (str = google.com.au)

The script isn't perfect; I saw things like mysearch.myway.com get by but it's good enough for general use. If you'd like to see the entire domain, just remove the 4 line chunk marked "Optional."

To call this using SQL, modify this sample script to suite your environment:

SELECT COUNT(*) as theCount, dbo.parseURL(referer) as referer FROM hitcounter
WHERE referer IS NOT NULL
GROUP BY dbo.parsedomain(referer)
ORDER BY thecount DESC

Your results should look something like this

11831 google.com
10542 yahoo.com
9101 msn.com
746 google.ca
624 google.co.uk

Note: NULLs aren't parsed and thsu won't kill this function..they'll just show up as NULL.

5 Comments   -
  • Comment by cm | March 22, 2007 @ 8:51 pm

    thanks -works great

  • Comment by Eric P | October 4, 2007 @ 5:50 am

    THis is absolutely fantastic! I am getting a curious result though, and I am not sure what to make of it.

    if I pass the URL from my logs through the function, it is successful at returning the root domain, unless the string was going to the web root.

    For example, If the entered string was http://www.google.com, the function returns a null, if the string was http://www.google.com/calendar?tab=mc , it properly returns google.com. Is the missing slash at the end of the input string the culprit?

  • Comment by Eric P | October 4, 2007 @ 10:14 pm

    Never mind...I found that for whatever reason, my logging software is adding scads of whitespace to the right of the site if the entry goes directly to the root website causing the record to exceed the allowed length of the variable. If I use an rtrim(site), everything works as it should. I wll be contacting the programmers of the logging software.

    Thanks a million for this!!

  • Comment by Edward | January 31, 2008 @ 5:33 am

    Thank you very much!

  • Comment by Jerry @ Moonvalley Software | June 6, 2008 @ 11:20 pm

    Thanks for the tips and the sample, very helpful.

RSS feed for comments on this post. TrackBack URL

Leave your comment