Keith Devens .com

RSS auto-discovery with PHP

I'm in the process of writing my own RSS aggregator. Naturally, I wanted to be able to use the new RSS auto-discovery method which has evolved over the past few days. Mark Pilgrim made some Javascript bookmarklets and a Python implementation to do this, but I needed a PHP implementation, so I wrote one . First the code, and then I'll explain:


<?php
function getRSSLocation($html, $location){
    if(!$html or !$location){
        return false;
    }else{
        #search through the HTML, save all <link> tags
        # and store each link's attributes in an associative array
        preg_match_all('/<link\s+(.*?)\s*\/?>/si', $html, $matches);
        $links = $matches[1];
        $final_links = array();
        $link_count = count($links);
        for($n=0; $n<$link_count; $n++){
            $attributes = preg_split('/\s+/s', $links[$n]);
            foreach($attributes as $attribute){
                $att = preg_split('/\s*=\s*/s', $attribute, 2);
                if(isset($att[1])){
                    $att[1] = preg_replace('/([\'"]?)(.*)\1/', '$2', $att[1]);
                    $final_link[strtolower($att[0])] = $att[1];
                }
            }
            $final_links[$n] = $final_link;
        }
        #now figure out which one points to the RSS file
        for($n=0; $n<$link_count; $n++){
            if(strtolower($final_links[$n]['rel']) == 'alternate'){
                if(strtolower($final_links[$n]['type']) == 'application/rss+xml'){
                    $href = $final_links[$n]['href'];
                }
                if(!$href and strtolower($final_links[$n]['type']) == 'text/xml'){
                    #kludge to make the first version of this still work
                    $href = $final_links[$n]['href'];
                }
                if($href){
                    if(strstr($href, "http://") !== false){ #if it's absolute
                        $full_url = $href;
                    }else{ #otherwise, 'absolutize' it
                        $url_parts = parse_url($location);
                        #only made it work for http:// links. Any problem with this?
                        $full_url = "http://$url_parts[host]";
                        if(isset($url_parts['port'])){
                            $full_url .= ":$url_parts[port]";
                        }
                        if($href{0} != '/'){ #it's a relative link on the domain
                            $full_url .= dirname($url_parts['path']);
                            if(substr($full_url, -1) != '/'){
                                #if the last character isn't a '/', add it
                                $full_url .= '/';
                            }
                        }
                        $full_url .= $href;
                    }
                    return $full_url;
                }
            }
        }
        return false;
    }
}
?>

The function takes two arguments. The raw HTML and the location you got it from. The location has to be there so relative links to the RSS file can be resolved.

First, PHP doesn't have an SGML parser built in like Python seems to, so I had to get all the <link> tags myself. A few regular expressions and some simple string splitting made it all real easy. Next, you cycle through all the links found, figure out which one is the RSS file, resolve it to an absolute URL if necessary, and return it.

I'll gladly take any suggestions for improving the code, and if I change it I'll update it here. Hope this helps.

Update: if you need a function to retrieve the HTML for you, feel free to use mine.


<?php
function getFile($location){
    $ch = curl_init($location);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Connection: close'));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_TIMEOUT, 15);
    $response = curl_exec($ch);
    curl_close($ch);
    return $response;
}
?>

This uses PHP's interface to the excellent cURL library, so this requires PHP 4.02 or higher.

Update: cool, mentioned on Scripting News! But oy! My name is spelled wrong. Later: he's fixed it, thanks Dave! Also seen on dive into mark and Keith Gaughan.

Update (6/6/02): Fixed a small bug where the RSS location returned would be incorrect if the RSS location had a query string with equals signs.


$att = preg_split('/\s*=\s*/s', $attribute);

had to be


$att = preg_split('/\s*=\s*/s', $attribute, 2);

so that it would only return up to two values, and not split on any more equals signs. Found while retrieving the RSS feed for Langreiter.com .

Update (6/19/02): small improvement thanks to Pepino.
Update (7/26/02): added a timeout to the getFile function.
Update (4/15/03): made it work with error_reporting(E_ALL), and changed the way it resolves relative URLs (see comments for more info).
Update (9/15/03): Note: this code is public domain.

Permalink | 21 Comments | Tags: Programming | id 2130

04:42 (utc-4)
edited: 2003-09-15 12:36 (utc-4)

Comments

I tried this with no luck

$location = 'http://www.phpheaven.net/';
$html = getFile($location);
getRSSLocation($html, $location);

-------------------------
I'm struggling to get this to work?
I know its more down to my php confusion with calling
functions, like Max above.

Any help would be appreciated.
As my searches for Auto Discovery bring me back to this
page time and time again.

-------------------------
Does the above code make a .php page "Auto Discovered"
so to speak.
How do i include it in my php?
An example would be much appreciated.

My page that i would like "Auto Discovery" is php?
I can using the default tags on a html page,
get auto discovery to work.
But i can not get a php page to Auto discover !

I'm newbie / noivice level..

Thanks in advance Mark.

I am sorry Keith Devens, but I had to modify the function to get all the feeds on the page. This is the code:


function getRSSLocation($html, $location){
    if(!$html or !$location){
        return false;
    }else{
        #search through the HTML, save all <link> tags
        # and store each link's attributes in an associative array
        preg_match_all('/<link\s+(.*?)\s*\/?>/si', $html, $matches);
        $links = $matches[1];
        $final_links = array();
        $link_count = count($links);
        for($n=0; $n<$link_count; $n++){
            $attributes = preg_split('/\s+/s', $links[$n]);
            foreach($attributes as $attribute){
                $att = preg_split('/\s*=\s*/s', $attribute, 2);
                if(isset($att[1])){
                    $att[1] = preg_replace('/([\'"]?)(.*)\1/', '$2', $att[1]);
                    $final_link[strtolower($att[0])] = $att[1];
                }
            }
            $final_links[$n] = $final_link;
        }
        #now figure out which one points to the RSS file
        for($n=0; $n<$link_count; $n++){
            if(strtolower($final_links[$n]['rel']) == 'alternate'){
                if(strtolower($final_links[$n]['type']) == 'application/rss+xml'){
                    $href = $final_links[$n]['href'];
                }
                if(!$href and strtolower($final_links[$n]['type']) == 'text/xml'){
                    #kludge to make the first version of this still work
                    $href = $final_links[$n]['href'];
                }
                if($href){
                    if(strstr($href, "http://") !== false){ #if it's absolute
                        $full_url[] = $href;
                    }else{ #otherwise, 'absolutize' it
                        $url_parts = parse_url($location);
                        #only made it work for http:// links. Any problem with this?
                        $full_url[] = "http://$url_parts[host]";
                        if(isset($url_parts['port'])){
                            $full_url[count($full_url)-1] .= ":$url_parts[port]";
                        }
                        if($href{0} != '/'){ #it's a relative link on the domain
                            $full_url[count($full_url)-1] .= dirname($url_parts['path']);
                            if(substr($full_url[count($full_url)-1], -1) != '/'){
                                #if the last character isn't a '/', add it
                                $full_url[count($full_url)-1] .= '/';
                            }
                        }
                        $full_url[count($full_url)-1] .= $href;
                    }
                    //return $full_url;
                }
            }
        }
        if (isset($full_url)) {
            return $full_url;
        } else {
            return false;
        }
    }
}

Monday, June 3, 2002

RSS auto-discovery with PHP

Comments