Infogami Developer Tutorial

The Open Library interface is powered by infogami -- a cleaner, simpler alternative to other wiki applications. But unlike other wikis, Infogami has the flexibility to handle different classes of data. Most wikis let you store unstructured pages -- big blocks of text. Infogami lets you store structured data. [1]

In addition to this, infogami facilitates the creation of dynamic HTML templates and macros. This flexible environment enables users to create, share and build collaborative interfaces. With Open Library in particular, we are focused on building a productive and vital community focused on the discovery of books.

In this document, you'll learn how to develop for infogami, including building new templates for displaying your own data, running your own copy, and developing new features and plugins.

To discuss this document, join the General Discussion mailing list.

  Summary (Audience Statement)

If you do not yet know Python, you should first study the
Python documentation or the free book
Dive into Python . Python is an easy language to learn, but
the OL codebase is probably not understandable by complete beginners.
For web server principles, the somewhat dated Philip and Alex's Guide to
Web Publishing is still an
informative read, though maybe someone can suggest something newer. You
should also understand the principles of software security --
see David A Wheeler's page for many documents.

2) A user or web designer wanting to improve or customize the Open Library's user interface, either for yourself or for our whole community. You will mainly want to study the section about template programming. You will need to know how to write HTML and it will help if you've done some server-side template programming (such as in PHP). It will also help if you've had some exposure to Python, but the programming skills you'll need for template writing are generally less intense than they'd be for extension writing.

3) A general user just wanting to know how the software works behind the scenes. You might not understand all the details, but reading the doc should give you a general impression of how sites like this are put together.

4) A librarian or metadata maintainer wanting to process large volumes of metadata for import into the Open Library. If you only want to import a few books, it's probably easiest to use the web interface (or the Z39.50 interface once we have one). To import bulk data, you'll have to process it into a format that Open Library can understand, which may require programming, but you can use your own choices of languages and platforms for that purpose since you only have to create uploadable files, rather than use language-specific interfaces. You'll mainly want to look at the section about data formats and schemas.

If you just want to be an OL user accessing or editing book data, you do NOT need to read this doc. The doc is about how to customize and extend the software, not how to use it. As developers and designers, our goal is to make the site self-explanatory for users and not need much separate documentation, but we do have some user docs at (url's....).

  Introduction

Infogami is a wiki application framework built on web.py. Actual applications (like Open Library) are written by extending Infogami through two layers: plugins and templates. Plugins are Python modules that get loaded into Infogami through a special API. They are invoked by submitting HTTP requests to the application, either HTML form posts or direct GET requests. Plugins can use any library or application code that they wish, and they create Python objects to represent results, that then get expanded to HTML by templates. Templates are a mixture of HTML text and user-written code, approximately in the spirit of PHP templates. The user-written code is in a special-purpose scripting language that is approximately a Python subset, which runs in a hopefully-secure server-side interpreter (embedded in the Python app) that has limited access to system functions and resources.

  Architecture

1.  Web Server

The lighttpd HTTP server runs Infogami through a FastCGI interface using Flup. (There can be multiple concurrent infogami instances that the lighttpd server distributes requests between, although Open Library currently just runs one.) Infogami is written in Python (Open Library currently requires 2.5 or greater) and uses web.py and ThingDB. ThingDB uses PostgreSQL as its data store. Psycopg2 is the Python driver for PostgreSQL. Open Library uses supervise (see also daemontools) to make sure everything keeps running. [2]

2.  Templates

The infogami application relies on various Web templates (these are code+HTML snippets). The initial templates are static files but they get edited through the wiki interface, and new ones get added through the wiki, so the real versions live entirely in the database.

2.1  web.py

The goal of web.py is to build the ideal way to make web apps. web.py allows you to build HTTP responses, makes the database easier to use, and creates a templating evironment that tries to bring Python into HTML. [*]

2.2  infogami

Each infogami page (i.e. something with a URL) has an associated type. Each type contains a schema that states what fields can be used with it and what format those fields are in. Those are used to generate view and edit templates which can then be further customized as a particular type requires.

2.3  OL application

[ See State of the UI ] ... [3]

3.  Search

Infogami also accepts plug-ins and Open Library uses one for the Solr search engine. Solr is a JSP currently sitting in a Jetty http server, so it communicates with Infogami through a local http socket. Solr itself wraps the Lucene search library. These run under Java (Open Library currently uses Java 1.5). Solr is built under Apache Ant and has a few config and schema files, plus a startup script (solr.sh) that has to be manually edited to set the port number. I think we currently use Lucene as a downloaded .jar file so we don't build it.

The solr-infogami plugin also calls out to an archive.org PHP script that expands basic search queries to advanced queries. It may also start using the openlibrary.org flipbook (with some possible customizations) to display OCA scans for pages containing fulltext search results.

4.  Data

Open Library has a bunch of catalog data and fulltext acquired from various sources, either sitting in the Archive or to be uploaded to there. I think the acquisition processes (including web crawling scripts for some of the data) is outside the scope of an Open Library software install. There are a bunch of additional scripts to make the stuff usable in openlibrary and these need to be documented. These include TDB Conversion Scripts written by dbg, and (for OCA fulltext) Archive Spidering and Solr Importing scripts written by phr.

  Template Basics

5.  Types

There are two kinds of types in infogami. Primitive types and compound types. Primitive types are for representing integers, strings etc.

Other primitive and compound types can be added by creating a page in the wiki with type type/type.

Sometimes you might want to link two different types. For example, book type has an author property and you want to add a books field to the author, which contains all books where author property of the book is this author. This is achieved by adding a backreference to the author type.

5.1  Primitive Types

Available primitive types are:

type/int
type/boolean
type/string - for single line text
type/text - for multiline text

5.2  Compound Types

Compound types specify what properties a thing of that type must have. Available compound types are:

type/type
Type for representing all types. It defines the following properties ...

• description of type type/text - description of type
• is_multiple of type type/boolean - to distinguish between primitive and compound types
• properties of type type/property - set of properties a thing of this type can have. This property can have multiple values.
• backreferenes of type type/backreference - set of backreferences a thing of this type can have. This property can have multiple values. Back-references are explained later in this document.

type/property
Type for representing properties in a type. It defines the following properties ...

• name of type type/string - name of the property
• type of type type/type - expected type of the value
• is_unique of type type/boolean - can this type have multiple values?
• description of type type/string - readable description of this property

type/backreference
later ...

type/page
Type for wiki pages. It defines the following properties ...

• title of type type/string - title of the page
• body of type type/text - body of the page

type/template
Type for template pages. It defines the following properties ...

• title of type type/string - title of the template
• body of type type/text - body of the template

6.  Templates

Infogami uses templates to control the look and feel of the site. Custom templates can be provided in the wiki to enhance functionality or customize the look and feel of the site.

There are two kinds of templates. Regular templates and type-specific templates. Type-specific templates are used to render a wiki page based on its type.

Regular templates must be placed at path templates/ and end with .tmpl extension. type-specific templates must be placed at type/ and end with .tmpl extension.

With Open Library, only site administrators can modify the templates that reside in the templates/ and type/ directories. If users want to override these templates, they must create their own set of templates in their user directory, and change their template root in their user preferences. For more information, see Custom Templates.

6.1  Type-specific Templates

There can be 4 type-specific templates for every type.

• view.tmpl - Template to specify how to view a page
• edit.tmpl - Template to specify how to edit a page
• repr.tmpl - Template to specify how objects of this type appear when viewing pages of other types which has a property of this type.
• input.tmpl - Template to specify how objects of this type appear when editing pages of other types which has a property of this type.

For more information on how the repr and input templates are used, please see Useful Functions.

6.2  Default Templates

It is not compulsory to specify all these templates for every type. When a type-specific template is not specified for a type, a default template is used. The following are the default templates:

• templates/default_view.tmpl - View Code or View Page Using Template
• templates/default_edit.tmpl - View Code or View Page Using Template
• templates/default_repr.tmpl - View Code
• templates/default_input.tmpl - View Code
• Note to Anand: what about http://demo.openlibrary.org:8080/templates/default_ref.tmpl ?

There are many other interesting regular templates. Some of them are:

• templates/site.tmpl - to specify the overall look and feel of a web site - View Code or View Page Using Template
• templates/diff.tmpl - to display a diff between two versions of a page - View Code or View Page Using Template
• ...

6.3  Custom Templates

A user can create their own set of templates by creating a subdirectory in the 'user' directory ('user/[username]'). A user can create a new template for each set of templates they want to override. If a template does not exist in a user's subdirectory, then the default set of templates will be used. Templates are created in the same manner as any new page is created in a wiki.

For purposes of illustration, we will view the contents of ('user/brewster/*')

• user/brewster/type/
• user/brewster/templates/

To view the Open Library using a set of custom templates, you must change your template preferences:

• Log in with your username and password
• Click on "Preferences" in the upper-right corner of the screen
• Change your template root to "user/[username]"

NOTE: Changing the template root to 'user/brewster' will load the templates used in this example.

It is also possible to create custom Cascading Style Sheet (CSS) files and JavaScript files in the wiki.

To create a custom CSS file in the wiki, create a page in your user directory entitled 'user/[username]/[filename].css' and select 'rawtext' for the pagetype. Once the CSS file has been successfully created, it can be linked to the template using the '$add_stylesheet' command:

JavaScript files can be created in the same manner, using a '.js' extension and the '$add_javascript' command:

It is important to note that when a page with the type 'rawtext' is added to the wiki, the editing function is not readily apparent. To edit a rawtext page, simply add '?m=edit' to the url.

Example:

• http://demo.openlibrary.org/user/anand/master.css
• http://demo.openlibrary.org/user/anand/master.css?m=edit

6.4  Useful Functions

There are some useful functions, which are very handy in writing view and edit templates.

• thingrepr(value, type) - This calls repr template of the specified type with value as argument.
• thinginput(type, name, value) - This calls input template of the specified type with value, name as arguments.

Lets take an example. Suppose there are many types, which need to display an image. Probably they will do that by defining a type type/image for representing images. Instead of worrying how to display and edit (upload) images, these types can just use thingrepr and thinginput macros and define repr and input templates for type/image. Later in future, if you want to change how to display/upload image, you don't have to change in multiple places.

There are some global functions, which are accessible by every template. For more information see Infogami API > Decorators.

7.  Macros

Macros are like functions in any programming language. Macros look very much like templates, but they can be called from wikipages and other templates and macros. Macros can be written in the wiki just like templates. Macros must be placed in macros/ path and type must be type/macro. As a convention, they are named in CamelCase.

7.1  Best Practices

Macros introduce an element of modularity when designing web templates in infogami. A common practice is to use macros for any repeatable design elements. This can be as simple as an HTML callout box, or a script. For instance, the "SAVE", "PREVIEW", and "DELETE" buttons on every edit form on the Open Library site are generated by the "EditButtons" Macro code below:

Intelligent use of macros results in clean, readable code that can be easily understood and repurposed by other users. For instance, the author view template is a good example of a template with a highly modular and streamlined design. View Page Using Template

7.2  Embedding

Macros can be invoked either through the wiki interface when editing a page in Markdown, or as a script when building web templates.

The syntax to call a macro from a wiki page is:

The syntax to call a macro from templates and macros is:

  WikiLanguage

8  Formatting Conventions

8.1  Page Headers

The edit and view templates are defined by two lines of code that must be at the top of the template file in order for them to function properly (The absence of either of these two lines will generate an error). For instance, this example is from the default edition template:

Additional variables can be controlled in these commands. For instance, in an edit template, the preview function can be disabled as follows:

Display of the <TITLE> of the page can be constructed by making a call to a universal language file. For instance:

would display "edit Tom Sawyer" in the edit view of an edition page by accessing the verbiage in the language file for '$_.EDIT' (edit) and the 'title' key in the 'edition' type [ See http://demo.openlibrary.org/type/edition/].

and:

would display "View Mark Twain" when viewing an author page by accessing the 'name' attribute in the 'author' type [See http://demo.openlibrary.org/type/author/].

8.2  Format Commands

8.3  Working with Types

The page 'get' statements make a call to the document type to pull the corresponding information for display on the page. For instance the following table illustrates our current author type:

description
is_primitive
properties	website of type type/string
properties	bio of type type/string
properties	photograph of type type/image
properties	name of type type/string
properties	death_date of type type/string
properties	alt_names of type type/string
properties	location of type type/string
properties	birth_date of type type/string
backreferences	Back-reference to authors property of type/edition

Therefore using this command:

on the view template will display the author name wherever used on an author template.

9  Template Syntax

9.1  Conditionals

'If' statements are used in much the same manner as in other scripting languages, invoking an action if a certain state exists. For instance, if page preview is activated in an edit template as follows:

the following command would format the HTML contents of the page wherever it is inserted in the edit template:

The 'if' statement can also be used in conjunction with the 'get' statement to hide empty data fields on a page. For instance:

would display the language a book was published in if that data is available for that book, or:

would display the date of birth and date of death for a given author on the author view template if they exist.

9.2  Loops and Iterations

9.3  Regular Expressions

  The Infogami API

10.  Plugins

Functionality of infogami can be extended by adding new plugins to it. Plugin in infogami is a python package with the following directory structure.

• foo/
• foo/__init__.py - required to make foo a python package
• foo/code.py - this is imported when infogami starts
• foo/templates/ - place to keep templates. This is optional.

see sample_run.py in infogami repository to find out how to add a plugin to infogami.

10.1  Special Pages and Modes

A plugin can add special pages to infogami and add new modes to wiki pages. Special pages are paths which are treated differently than regular wiki pages. login, logout, sitepreferences etc. are some examples of special pages. Modes are actions that be done on a wiki page. view, edit, history, diff etc. are some examples of wiki modes.

infogami.utils.delegate module provides two special classes page and mode. Any class that extends from page becomes a special page with path same as the class name and any class that extends from mode becomes a wiki mode with name of the mode same as the class name.

An example:

 # hello/code.py
 class hellopage(delegate.page)
     def GET(self, site)
         return "hello page!"
class hello(delegate.mode)
     def GET(self, site, path)
         return "hello mode for " + path

If this hello plugin is added to infogami, /hellopage page displays hello page! and /foo?m=hello displays hello mode for foo.

10.2  Actions

A plugin can also add new actions to the system. Action is a utility function, which can be invoked from the command line as shown in the following example.

The default action is startserver and that is invoked when no action is specified.

To convert a function in to an action, it must decorated with infogami.action decorator. The following is an example of action.

Once this plugin is added to infogami, this action can be used.

The help action prints the list of all available actions.

There are some important actions, that every plugin author must know.

install: The install action runs all the install hooks. Install hook is a function without any arguments decorated by infogami.install_hook. Any plugin can add a new install hook to do some actions to initialize an infogami instance. Some of the actions are install hooks too.

movefiles: The movefiles action moves all the files in $plugin/files/ from every plugin to static/ directory. This is also an install hook. If a plugin needs to access any static resources, they must be put in files/ directory of that plugin. Keeping resources directly in static/ is not a good idea and should be discouraged.

movetemplates: The movetemplates action moves templates from every plugin to wiki. Optional prefix can be specified to limit the templates that are moved into the wiki. This is also an install hook.

10.3  Classes

10.4  Decorators

To make a function accessible in templates, that function must be decorated with infogami.utils.view.public decorator. Since templates can be written by untrusted users, utmost care must be taken when exposing a function in templates.

10.5  Security Considerations

The basic flow is: the user submits a url or HTML form; the FCGI handler matches the URL patch against a series of regexps to dispatch the request to the proper plugin; the plugin receives the GET or POST data and creates results as Python objects (e.g. a list of strings or ints); the plugin then loads a template (designated in the plugin code) which receives the result objects as arguments. The template code then expands the result to HTML which gets presented to the user's browser. Thus, a user can "re-skin" the application by modifying the static HTML or the script code in the templates that he or she uses. A few initial templates (such as the application's top-level HTML page) are included in the application source tree, and the rest are created and edited through the wiki interface.

Plugins are written by the application developers and are part of the source code tree for the application. Templates are entered and updated through the wiki HTTP interface, are stored in the database just like other wiki pages, and can be written by arbitary users. Thus, templates are potentially hostile and therefore require security precautions during execution. Templates give a flexible means of creating presentation HTML, but for security reasons, operations requiring access to system functions must be implemented in plugins, not templates. Templates can be application-wide or can be specific to a given user account.

  Troubleshooting