You are here: System » FeedPlugin

FeedPlugin

Syndication feed parser

This is a new plugin to render RSS and ATOM feeds from internet sites. It actually is a rework of the old HeadlinesPlugin now much leaner and more robust, i.e. facing unicode website feeds. Instead of implementing a RSS parser by its own FeedPlugin hands off duties to CPAN:XML::Feeed and only takes care of caching and integrating results on a Foswiki page.

Examples

List headlines at http://blog.foswiki.org; auto-discover news feed:

%STARTSECTION{"example1"}%
%FEED{
   "http://blog.foswiki.org" 
   discover="on"
   header="<div class='alt'>$n"
   footer="$n</div>"
}%
%ENDSECTION{"example1"}%

Format the most recent posting on the Foswiki Blog:

%STARTSECTION{"example2"}%
%FEED{
   "http://blog.foswiki.org/feed/"
   limit="1"
   header="<div class='alt'>$n"
   format="---+!! <div class='foswikiGrayText foswikiRight foswikiNormal'>$date</div> [[$link][$title]]
           $content"
   footer="$n</div>"
}%
%ENDSECTION{"example2"}%

04 May 2019
FORMATLIST documentation and examples

FORMATLIST is Foswiki's Swiss army knife for manipulating text. It has many parameters and using it can be sometimes perplexing. I hope you find my notes useful.

Table of contents

Syntax

%FORMATLIST{ "<list>" … }%
list can be any string or an INCLUDed topic . Double quotes must be escaped.

See the FORMATLIST documentation for details.

FORMATLIST has many parameters and using them can be perplexing. They come in three classes.

used on with single line
or split in items
used with items only used with patterns within items only
tokenize separator pattern
split lastseparator format
replace limit map
header skip  
footer sort  
null reverse  
hideempty unique  
  exclude  
  include  
  selection  
  marker  

The internal processing sequence

The behaviour of FORMATLIST is better understood when you know the internal processing sequence. Here is a summary.
1. map
The %map structure is built from the map="key1=value1,key2=value2, ..." parameter fo use in step 8.

2. tokenize the list
If tokenize="regex" is defined, every matched occurrence of the regex in the list is replaced by a token of the form token_n, where n is 1 .. infinity.

3. split the list
The list is split over the matched occurrence of regex in split="regex".
For example: list="a,b~c,d~e,f" and split="~" creates a three list items in array @list=('a,b', 'c,d', 'e,f')

4. replace text
Replace items in the list using the key value pairs provided in replace="key1=value1,key2=value2,...". The replace happens after the tokenize, so some items can be protected from the replace.

5. replace the tokens
Replace the tokens from step 2 with their original values so we are back to normal.

6. the list is sorted
Sort the list items if sort="on".
Note: the sort is performed on the unformatted list. The formatted list is not sorted. You can sort the formatted list using a second FORMATLIST macro.

7. reverse the list
Reverse the list order (last item at the top) if reverse="on"

8. loop
steps 8a to 8h are performed individually for each item in the list

9. create the list result
The result list is created by joining the items over the separator text provided in parameter separator="separator text"
If the parameter last separator="last separator text" is provided, the last item is joined preceded by last separator text

10. create the replacement text
The replacement text is created using the header text and footer text provided in the header and footer parameters.
replacement text is the concatenation of header text, result and footer text.

8a ignore "invalid" list items
Invalid list items are determined by the following criteria:
items matching the regex in parameter exclude="regex"
items NOT matching the regex in parameter include="regex"
items that are empty ('')

8b skip valid items
ignore the first n (provided in parameter skip="n" valid items for processing and count

8c limit processing
limit the processing to the next n (provided in parameter limit="n" valid items for processing

8d match the pattern
match the list item to the regex provided in parameter pattern="regex"

8e format the result
reformat the item using the format rules provided in parameter format="format rules"
At this point the map created in step 2 is used to expand the $map(key) format variable in the format rules.

8f check for uniqueness
If parameter unique="on" the resulting item is matched against all previous results. If a match is found, the result is discarded.

8g mark selected lines
Match the item to the regex provided in parameter selection="regex". If a match occurs, the marker $marker in the format rules is replaced by the marker text provided in parameter marker="marker text"

8h ignore empty items
Ignore empty items if parameter hideempty="on"

Examples

the list

The following list is used as input for all examples below.
  • Set theList = d~e~fisherman~~j~fishingrod~c~~a~fish~i~~g~fishbowl~l

replace

Replace fish with shark

%theList%


%STARTSECTION{ "replace" }%
%FORMATLIST{ 
    "%theList%"
    replace="fish=shark"
}%
%ENDSECTION{ "replace" }%

tokenize, replace

Replace fish with shark, but protect fisherman and fishingrod by tokenizing them.

%theList%


%STARTSECTION{ "tokenize" }%
%FORMATLIST{ 
    "%theList%"
    tokenize="fish(i|e)[^~]*"
    replace="fish=shark"
}%
%ENDSECTION{ "tokenize" }%

sort

Sort sorts the list items in alphabetical order

%theList%


%STARTSECTION{ "sort" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    sort="on"
}%
%ENDSECTION{ "sort" }%

replace sort

Sort sorts the list items in alphabetical order, taking into account the replacements. Note that the replacements occur anywhere in the list items. For instance, the a in fisherman is replaced by a z to read fishermzn.

%theList%


%STARTSECTION{ "replacesort" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    replace="a=z,d=y,g=x,j=w"
    sort="on"
}%
%ENDSECTION{ "replacesort" }%

pattern

Pattern can be used to reformat the list into a table.


%STARTSECTION{ "pattern" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    pattern="([^~]+)~([^~]+)~([^~]+)"
    format="|$3|$2|$1|"
    separator="$n"
}%
%ENDSECTION{ "pattern" }%

pattern sort

Notice that the sort order is that of the original list, not the formatted list.


%STARTSECTION{ "patternsort" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    pattern="([^~]+)~([^~]+)~([^~]+)"
    format="|$3|$2|$1|"
    separator="$n"
    sort="on"
}%
%ENDSECTION{ "patternsort" }%

selection marker

Items in the list can be marked providing an appropriate format. There is no need for a pattern parameter. The default (.*) will do.

%theList%


%STARTSECTION{ "selectionmarker" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    format="|$1|$marker|"
    separator="$n"
    selection="man"
    marker="marking a male"
}%
%ENDSECTION{ "selectionmarker" }%

map

The map parameter is a convenient shortcut, avoiding the more cumbersome IF{ "\"$1='a'\" then=\"alpha\" else=\"$1\"" }: Missing operator in '"$1='a'" then="alpha" else="$1"' .


%STARTSECTION{ "map" }%
%FORMATLIST{ 
    "%theList%"
    split="~~"
    pattern="([^~]+)~([^~]+)~([^~]+)"
    format="|$map($3)|$map($2)|$1|"
    separator="$n"
    map="a=*alpha*,e=*epsilon*,i=*iota*"
}%
%ENDSECTION{ "map" }%

Tags: Examples, FilterPlugin

Syntax

Parameter Description Default
"..." or href="..." source url; this can either be a direct link to the RSS/ATOM feed or to the website serving the feed in which case you need to enable discover  
expire="..." or refresh="..." refresh rate for caching the feed; this can be specified using an expiry term such as 1 d for one day or 1 h for one hour etc specified in $Foswiki::cfg{FeedPlugin}{CacheExpire}, defaults to 1 d
limit="..." maximum items to show 0 (no limit)
skip="..." number of items in the feed to skip showing the rest 0
header="..." format string to be prepended to the list of items in the feed  
format="..." format string for each item on a feed * [[$link][$title]]
footer="..." format string to be appended to the list of items in the feed  
separator="..." format string to separte items in the feed  
discover="on/off" switch on feed discovery starting at the source url off

An empty result will be returned when no items have been found in the feed (or all items have been skipped).

The format parameter may contain the following variables expanding to respective properties of a feed item:

  • $author
  • $base
  • $category
  • $content
  • $id
  • $index
  • $issued, $issued(<date-format>)
  • $date, $date(<date-format>)
  • $link
  • $modified, $modified($lt;date-format>)
  • $summary
  • $tags
  • $title

In addition the header, format, separator and footer format strings may contain:

  • $feed_author
  • $feed_base
  • $feed_copyright
  • $feed_format
  • $feed_generator
  • $feed_language
  • $feed_link
  • $feed_modified, $feed_modified($lt;date-format>)
  • $feed_tagline
  • $feed_title

… as well as the standard escapes:

  • $percnt
  • $dollar
  • $n

Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. "Extensions Operation and Maintenance" Tab → "Install, Update or Remove extensions" Tab. Click the "Search for Extensions" button. Enter part of the extension name or description and press search. Select the desired extension(s) and click install. If an extension is already installed, it will not show up in the search results.

You can also install from the shell by running the extension installer as the web server user: (Be sure to run as the webserver user, not as root!)
cd /path/to/foswiki
perl tools/extension_installer <NameOfExtension> install

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See https://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Dependencies

NameVersionDescription
XML::Feed>=0.53Required.
LWP::UserAgent>=0Required.
Cache::Cache>=0Required.
Digest::MD5>=0Required.
URI>=0Required.

Change History

28 May 2018 support $date as a synonym for $issued, added expire parameter (basically an alias for refresh but more intuitive); supports Foswiki's standard proxy/noproxy settings
24 Apr 2016 fixed docu; fixed discover mode; added support for non-unicode Foswiki engines
18 Mar 2016 be more robust on feeds not publishing proper dates
16 Mar 2016 initial release
This site is powered by FoswikiCopyright © by the contributing authors. All material on this site is the property of the contributing authors.
Ideas, requests, problems regarding Wiki? Send feedback