DCat - a Web Application Server in the D Programming Language.
BEV Home Page,/|Contact,mailto:steve.teale@britseyeview.com|Comment,,#TFC-toggler|Adia's Place,/ap/|Software,/software/


Please note that what is described here is very much 'work in progress', internally inconsistent, and subject to the usual caveats as to suitability for any particular purpose.

However, you may download the code, sample files, and a Windows executable, in its current state (as of 20 April 2009).

Contents.

  Overview

What is DCat

Application Sets

Applications

Pages

More Detail

Flow of control in a DCat application

Page sequencing constructs

Template processing options

Application state processing sequence

Default Directory Structure

ASD file reference

Notation

ApplicationSet Tag

Application Tag

Error Tag

Exit Tag

MySQL Tag

Namespaces Tag

Page Tag

PIOPath tag

Template Tag

Variable Tag

XML Tag

Page Templates

Template page syntax

Run time representation of templates

Compiled template file format

PIO Interface

The Interface

The returnProc method

The preamble method

The interpretIP method

The doInsertions method

The generateHeaders method

The generateContent method

The generateCookies method

The postamble method

PIO exceptions

Communication between PIO stages

Cacheing

What is cached.

Logging

Log4D.

Creating a PIO

Boilerplate implemntation.

Caveats on dynamic loading of PIOs. Building and Installing DCat

Building.

Configuring Apache.

Testing.
Overview.
- What is DCat.

DCat is a compact web application server written in the D programing language. It provides an environment in which complex web applications that require dynamic generation of web content can be designed, built and executed. DCat applications are defined using declarative XML, and implemented by dynamically loaded modules written in the D programming language.

DCat requires sets of related web applications to be described in an XML Application Set Description - ASD. Such a description specifies what applications constitute the set, and what HTML pages or other server responses constitute each application. The flow of control between these pages is also specified in the ASD, with facilities for conditional branching, looping, etc. Variables can be defined in the ASD at application-set, application, or page scope, so that business data can be kept out of the programming arena, and can be easily changed at a central point. The ASD allows much of the high-level structure and logic of a complex system to be defined and documented before a single page is designed, or any custom logic is programmed.

Pages can also refer to Page Implementation Objects - PIOs. PIOs are D modules that generate web responses, whose attributes and behavior are controlled by information in the ASD.

As in a Java Servlet Container, DCat can parse and compile template .dcat files to produce run-time representations that can generate web responses. A D module implementing an appropriate interface can also be used like a Java servlet.

Compiled DSP modules - object files - are dynamically loaded as required and cached as specified in the ASD, or by default, indefinitely.

State information may be embedded in individual web pages as XML contained in hidden form fields or cookies, and Dcat provides a Javascript TinyXML parser to minimize the pain of cross-browser client side coding if this is desired. The XML state data is returned to the server side as post or query-string data, or as a cookie, as specified in the ASD.

DCat acts in cooperation with the Apache web server in much the same way as Tomcat cooperates for the Java servlet environment, using the AJP13 protocol. However DCat does not attempt to emulate TomCat in its capability to act as a stand-alone web server. Apache already performs that role perfectly well. It is envisioned that to avoid dependence on some particular intermediary, connectors for Dcat could also be provided to use other mechanisms such as FastCGI.
- Application Sets.

Dcat recognizes that most web facilities consist of a set of related applications. The individual applications may be discrete, or may communicate, either directly through URLs, or through a shared database. A single application set might well constitute the entire role of some web server. However, the notion of an application set will often also be convenient for the administrative separation of responsibility for development, maintenance, and operation of applications. Currently, the Dcat hierarchy consists of the Application Set, which is composed of Applications, which are in turn composed of Pages. The pages don't have to be HTML, they just represent individual responses from the web server. If they are HTML, they will probably have embedded XML elements to carry data from the server to the client side and to return data to the server.

At the application set level, it's possible to define variables that apply to the collection of applications, and the destination of log/debug information for the application set. It’s also possible to set a variety of defaults that can be overridden as required at the application level, such as connection strings for databases, and various paths. For example:
<?xml version="1.0"?>
<ApplicationSet verbosity="debug" logpath="/dev/pts/2">
   <Database name="orders" connect="connection string"/>
   <Database name="audit" connect="another connection string"/>
   <Error path="/AppSetName/errpage.htm"/>
   <Exit path="/AppSetName/thankyou.htm"
   <PIOPath path="/AppSetName/PIO"
   <Variable name="varname1" type="string" value="varvalue1"/>
   <Variable name="varname2" type="int" value="42"/>
   ...
</ApplicationSet>
Most of these things can be overridden by Applications defined within the Application Set.

- Applications.

An application defines a set of Pages. It can override information specified at the application-set level. It can also specify variables in the scope of the application. These are in a different namespace than the Application-Set variables. For example:
<?xml version="1.0"?>
<ApplicationSet name="AppSetName" verbosity="error" logpath="/dev/pts/2">
   <Database name="orders" connect="connection string"/>
   <Database name="audit" connect="another connection string"/>
   <Exit path="/AppSetName/thankyou.htm" />
   <Error path="/AppSetName/errpage.htm"/>
   <Variable name="orgname" value="ACME Industries Inc"/>
   <Variable name="copyright" value="Copyright (C) 2000 - 2009"/>
   <Application name="app1" state="hxml" xmlvar="__DCat_xml">
      <Exit path="/AppSetName/AppName/thankyou.html"
      <Error path="/AppsetName/Appname/errpage.html"/>
      <PIOPath path="/AppsetName/Appname/bin"/>
      <Database name="oldorders" connect="yet another connection string"/>
      <Variable name="MaxQueryRows" type="string" value="50"/>
      <Variable name="Version" type="string" value="1.9.1"/>
      <Page ... />
      <Page ... />
   </Application>
   ...
</ApplicationSet>
- Pages.

An application comprises one or more pages, each of which will require a DCat template and/or some PIO - code to generate content. Pages have attributes that determine whether a template will be used, and which of a standard set of methods they will utilize from the PIO. A page may also declare itself to be the start page for its parent application

Pages also describe a path or paths through the application by specifying the page that is to follow them, possibly subject to dynamically determined conditions. Because this conditional flow control is provided at the page level, an Application definition constitutes a high-level program.

Page scope variables can also be defined.
<?xml version="1.0"?>
<ApplicationSet name="AppSetName" verbosity="error" logpath="/dev/pts/2">>
   <Database name="orders" connect="connection string"/>
   <Database name="audit" connect="another connection string"/>
   <Exit path="/AppSetName/thankyou.htm"
   <Error path="/AppSetName/errpage.htm"/>
   <Variable name="orgname" value="ACME Industries Inc"/>
   <Variable name="copyright" value="Copyright (C) 1999 - 2002"/>
   <Application name="app1" state="hxml" xmlvar="__Dcat_xml">
      <Exit path="/AppSetName/AppName/thankyou.htm"
                  error="/AppsetName/Appname/errpage.htm"/>
      <Database name="orders" connect="yet another connection string"/>
      <Variable name="MaxQueryRows" type="string" value="50"/>
      <Variable name="Version" type="string value="1.9.1"/>
      <Page name="page1" template="page1" phases="pci" options="si" next="+" >
         <Variable name="foo" value="bar" />
         <Variable name="quality" value="horrid" />
      </Page>
      <Page name="page2" template="page2.dsp" pio="page2.o" phases="pr" options=""
                            next="next? page1: page3" />
      <Page name="page3" phases="r" />
   </Application>
   ...
</ApplicationSet>
More Detail.
- Flow of control in a DCat application.

DCat generally expects its requests to be of the form:
http://www.DCatserver.com/dcat/appname/pagename
or
http://www.DCatserver.com/dcat/appname/pagename?somevar=somevalue& . . .
Here, "dcat" corresponds to a mod_jk JkMount directive in Apache's httpd.conf file, and the rest of the URI provides a path into the application set tree, the root of which is specified in the DCat service/daemon’s properties file.

The initial request takes the limited form:
http://www.DCatserver.com/dcat/appname
In this case the starting page is determined from the ASD.

These URIs make no attempt to specify what page should be processed next. They merely report what page submitted the request except in the case of the initial request, which just states what application is to be entered. The DCat framework will determine what page is to be served up next. You make those decisions when you write the XML for the application set. The ASD is parsed and loaded when the DCat service/daemon is started. Then the application-set data becomes available as read-only data to all worker threads in the application.

For cosmetic purposes, you may, as appropriate, add either of the pseudo page names "_start_" or "_nextpage_" to the URI. This provides information in the address bar that may make the sequencing of pages clearer to the user. For example, if the first two pages are "page1" and "page2", and the actual pages have similar headings, then when page1 submits, the address bar will show "/dcat/app1/page1", while the heading of the displayed page will be "Page 2". This is mildly confusing, and it is probably better to see "/dcat/app1/page1/_nextpage_" in the address bar. This makes it clear that the situation is not a contradiction.
[If browsers responded appropriately to a 201 response code, and used the URI from the location response header to populate the address bar, this stuff would not be neccessary.]. The following forms are therefore acceptable:
http://www.DCatserver.com/dcat/appname/pagename/_nextpage_
http://www.DCatserver.com/dcat/appname/_start_
When DCat receives such a request, the first thing it does is to parse out the variable values supplied in the GET/POST request. DCat makes no particular distinction between these two types of request, and simply places the variable name/value pairs in a symbol table in a 'Request Object' for future reference.

What happens next is determined initially by the 'phases' attribute on the page of the supplied name, or the page designated as first. If this contains the 'r' flag for return processing, then the PIO for the page is loaded and cached, and its returnproc() method is called (as long as this isn't the first call to the first page). If the PIO doesn't have the appropriate entry point, DCat reports an error and sends an e'ror page back to Apache.

Once any return processing has been completed, DCat determines from the page definition, using any supplied conditions, what page is to be served next. (The return processing can if necessary force an arbitrary page, but this is rather like using goto. You can't then tell what will happen simply by knowing the values of variables and looking at the ASD.)

A sequence of operations is then possible, depending on the value of the 'phases' attribute on this next page:
  1. Preamble (p) - general processing before the next page can be generated,
  2. If the page specifies a template - a .dcat file, check the modification date of the template against any existing translated version, and if necessary, repeat the translation process, or translate the .dcat file for the first time.
  3. If the phases attibute specifies insertion point processing (i), then the corresponding method in the PIO is called. Otherwise a standard proceedure is used for substitution.
  4. Set cookies (c) - set up the values for any required cookies,
  5. Extra headers (h) - set up the strings for any HTTP headers other than the standard ones,
  6. Load the compiled .fragments file and select its generate method, and call that to output the page content, or if there is no template, generate the content using the appropriate method in the PIO.
  7. Send the headers and content back to Apache via the connector.
  8. Call the postamble of any PIO if specified (a).
The .dcat file utilizes syntax similar to that of a .jsp file with the exception of referenced components. The compilation process for the .dcat file results in a .fragments file - a list of literal sections and substitution points that can be very quickly instantiated at run time. When it is 'finalized' by substitution, sources for substitution variable are:
  • Get/post request data symbol table,
  • Cookies symbol table,
  • State data,
  • Application set object variables.


These constitute six distinct namespaces, since the ApplicationSet object is essentially a set of nested string[string] associative array references that represent the content of the ASD XML file for a particular page - appset/app/page. In the .dcat file these are referenced via prefixes like AS:varname, AP:varname, PG:varname, CK:varname, ST:varname, and varname. The undecorated varname refers to the GET/POST data. The namespace prefixes can be overidden in the ASD file.

The interface implemented by PIO objects is as follows. Only a subset of these methods will need to be implemented in most cases.
struct DCatContext
{
   const(string[string]) requestdata;
   const(string[string]) cookies;
   const(TinyXML) state;
   const(ApplicationSet) appset;
   bool delegate(string objectName, Object object) holdObject;
   Object delegate(string objectName) retrieveObject;
   void delegate(string objectName) releaseObject;

}

interface PIO
{
   string returnProc(string prevpage, string nextpage, DCatContext* context);
   void preamble(DCatContext* context);
   string interpretIP(string ipname, DCatContext* context);
   string doInsertions(FragmentList() fl, DCatContext* context);
   string[] generateHeaders(DCatContext* context);
   string generateContent(DCatContext* context);
   string[] setCookies(DCatContext* context);
   void postamble(DCatContext* context);
}


When the response generation is complete, but before the end of processing is signaled to the Apache connector, a PIO postamble entry point may be called if specified for the page to give you an opportunity to clear up any data structures you may have created in the stages described above.

The page sent to the browser in this way will in turn submit a request embodying its qualified name (/appset/app/page), and the sequence will be repeated as required.

If a page that has no next-page specification is submitted, the exit page specified for the application, or failing that for the application set, will be displayed. In this case only, the framework will simply serve the contents of a file.

- Page sequencing constructs.

In the example above,

<Page name="page1" phases="pc" options="s" next="+"/>

page1 and page 2 are used in sequence, as indicated by the next="+" attribute on page1. The possible values for next have the following effects:

"."Repeat the same page
"+"Go to the next page as defined in the ASD
"-"Go to the previous page as defined in the ASD
"^"Go back to the first page of the application
"pageN"Go to the literally named page
"*var"Go to the page named by variable var
"?cond page1, page2"Ternary conditional, if variable cond is null or blank, page1, else page2
"#jump page2, page3, page4"Jump to a page depending on the integer zero based value of variable "option".


If there is no 'next' attribute, the next page will be the exit page specified for the application set, or for the particular application. It is generally assumed that this exit page does not submit a request referencing a page in the application. Nothing prohibits this, but such cases should be dealt with by having a page for that purpose within the application.

- Template processing options.

Most pages will use a .dcat template, but this is not required. If you choose to do so, you can generate the entire page, or everything after the HTTP headers. Where most of the page needs to be generated, skipping the overhead associated with template interpretation may result in improved performance.

If a template is required, you may choose to parse and interpret it as required, or to parse it once, and cache a more efficiently interpretable version or further use. Cached templates - the 'c' option - will give better performance, but will of course use more memory - the usual tradeoff.

These choices are made in the page options flags. The 'G' option indicates that you will generate everything; 'g' indicates that you will generate everything after the HTTP headers. In either case the generate() entry point in the PIO will be called once at the appropriate point to generate whatever was specified.

The 'i' option indicates that the page is to initialize the application state mechanism as described in the next section.

- Application State Processing Sequence.

DCat supports the use of XML data in your HTML pages for the maintenance of application state. You can specify that you want to do this by setting the 'state' attribute on the Application tag in the ASD to "hxml" (XML in a hidden HTML field) or "cxml" (XML in a cookie). The attributes of the application can also provide the name of an HTML field and DCat variable that is to be used to store the application state XML (the default name is __Dcat_xml).

When a page submits, the value of this field is examined. If it exists, and contains text, its value is parsed, and the results are loaded into a simple XML object model. This read/write object exists on the DCat request object, and is available to all phases of page processing.

When the page template is interpreted, the interpreter treats an occurrence of <%=__Dcat_xml%> (or whatever you specified as the name in the application definition) as a special case. At that point, it will generate packed XML (no extraneous whitespace, and appropriate encodings) from the XML object, and substitute the XML as the field value.

This mechanism requires your .dcat file to contain a hidden variable such as:



The XML variable name appears twice, once as the name of the HTML input field (from which XML from the page will be loaded), and again as the name of the substitution, so that the XML will be placed there during template processing. Note the single quotes around '<%=__Dcat_xml%>'. These are necessary since your XML will doubtless contain numerous double quote characters. Single quotes in the XML are encoded (").

It’s also worth noting that to prevent redundant submission of HTML form data when using XML, you should use two or more forms in your HTML document. One of these will be reserved for submission of the page, e.g.:
<form name=appstate action="/appset/app1/page1" method=POST>;
<input type=hidden name=__Dcat_xml value='<%=__Dcat_xml%>'>
<input type=submit value=" Submit ">;
</form>;
Note that the submit button(s) will probably have to be in this same form, since browsers appear to have difficulty submitting formA from a submit button in formB.

The initial representation of your application data should be named for the application - appname.xml - you can specify its location by using the XML tag in the ASD. It will be loaded, once per application invocation, by the page that has the 'i' option set. If the XML variable already contains data, this step is skipped - it's already been loaded, and the application has recycled to the same page.

- Default Directory Structure.

Although you can override most of it in the ASD file, DCat has an assumed default directory structure for its execution environment. For each instance of the service/daemon there will be a top-level directory named for an application set. Under this, the directory structure should be as follows:


Directory structure
There is a subdirectory under AppSetRoot for each application, and under each of these directories for individual pages that contain template files, and PIO objects.

The generic log file for the application set lives inder the application set root. Messages are written to that when there is no application context, e.g. during start up, or when forward-request AJP packets are received from the web server. Once a context is established - after the forward-request packet has been parsed, the application log in the application directory is used.

Generic error and exit page files can be placed in the application set root directory, and or in individual application directories.
ASD File Reference.
- Notation.

Tags are described here using a loose notational convention as follows:

Required attributeattname="attvalue"
Optional attribute[attname="attvalue"]
Enumerated attributesattname="cat | dog | mongoose"
Required child tag <tagname . . . />
One or more of same name<tagname . . . />+
Optional tag (zero or one)<tagname . . . />?
Zero or more tags of same name  <tagname . . . />*


Attribute values in italics are explained separately.
- ApplicationSet Tag.

The top level tag in a DCat ASD is the application set tag. It takes the following form:
<ApplicationSet logpath="/path/to/log/file"
                [name="AppSetName"] 
                [verbosity="error | warning | info | debug"]
                [version="versionstring"]
                [xmlvar="subst_here"]>
    <Application . . . />+
    <Database . . . />*
    <Error . . . />?
    <Exit . . . />?
    <MySQL . . . />?
    <Namespaces . . . />?
    <PIOPath . . ./>?
    <Template . . ./>?
    <Variable . . . />*
    <XML . . ./>?
</AppSetName>
Tags of any other name are ignored. You may include element text and comments for documentation.

If the name attribute is not present, it defaults to the first joint of the name of the file from which the xml file was loaded (e.g. AppSetName.asd -> name="AppSetName"). If the verbosity attribute is not present at the application-set scope, it defaults to error. If the version attribute is not present it defaults to "0.0". If the xmlvar attribute is not present, it defaults to "__DCat_xml".

- Application Tag.

There must be one or more Application tags, taking the form:
<Application name="AppName" [state="hxml | cxml | form"] [xmlvar="subst_here"]>
    <Database . . . />*
    <Error . . . />?
    <Exit . . . />?
    <Page . . . />+
    <PIOPath . . . />?
    <Template . . . />?
    <Variable . . . />*
</Application>
If the state attribute is not defined, it defaults to the empty string. If the xmlvar attribute is not defined it defaults to whatever is set for the application set.

Notice that it is possibly to overide or augment items that were defined at the Application Set level.
- Error tag.

The Error tag is optional. It takes the form:

<Error path="/path/to/error.html" />

The path provided will be used to construct an error page by appending error information and "" to the HTML file found there. It may be defined at application-set or at application scope. A definition at application scope overrides any value at application-set scope. This is a filesystem path. A path without a leading '/' will be taken as relative to the application set, e.g. path='path/to/error.html' in application-set 'trivia' will be taken as '/trivia/path/to/error.html', or if trivia is an alias, the corresponding path.

- Exit Tag.

The Exit tag is optional. It takes the form:

<Exit path="/path/to/exit.html"/>

The path provided will be used as a target URI when the application submits from a page for which no "next" attribute is provided. It may be defined at application-set scope. A definition at application scope overrides any value at application-set scope. The tag is optional, a DCat application that has no exit tag, and any page with no next attribute, will attempt to load a page "/appsetname/index.html". Note that the path here is absolute or relative to your web server root. A path without a leading '/' or "http://" will be taken as relative to the application set, e.g. path="path/to/error.html" in applicatio-set "trivia" will be taken as "/trivia/ path/to/error.html".
- MySQL Tag.

The MySQL tag is optional in the sense that you don't need one if your application(s) don't use a database. It has optional attributes at application-set scope:
<MySQL [none = "none"]
         host="hostname"
         name="database name"
         user="user name"
         epw="encoded password"
         [preload="Y | N"] />
The "none" attribute with any non-empty value e.g. none="none", overrides all other attributes here, and says this application makes no use of a database, even if one was specifies at application-set level. Specifying none will marginally increase performance. Otherwise, any of the non-optional attributes not defined at application-set scope must be provided at application scope. The epw attribute is a base64 encoded encryption of the specified user's database password. The DCat utility DCatpwe will encode passwords, and the DCat framework will decode them using the same key. You can set your own key by recompiling a small DCat module.

The database name may be a simple name, or a composite name represented by a string of the form:

"prefix+*varname+suffix"

In this case, the database name will be composed by looking up the value of variable varname, then prefixing prefix, and postfixing suffix. For example, the composite name

"registered_+*customer+_users"

when the variable customer has value "acme" will result in a database name of "registered_acme_users.

"+*customer+_users" or just "*customer+_users" will map to "acme_users", and so on.

The preload attribute is interpreted as boolean. A connection to this database connections will be established when the application starts, and DCat will report an error if it can't be opened at that time. If the database marked as preload has a composite name, the connection will be created as soon as the framework detects that the variable used is non-blank. If preload is not specified, your PIO code must establish the connection as and when required.

Tags for other databases may be introduced in future versions.
- Namespaces Tag.

The Namespaces tag may appear within a ApplicationSet or an Application. It is optional, and takes the following form:
<Namespaces    [appset="appsetNSQ"]
               [app="appNSQ"]
               [cookies="cookiesNSQ"]
               [page="pageNSQ"]
               [state="stateNSQ"] />
At substitution points in pages, where you want to use variables defined in the ASD, you must qualify the variable name with a namespace qualifier. The Namespaces tag allows you to choose your own qualifiers. By default they are:

Application set scope - AS
Application scope - AP
Page scope - PG
Cookies - CK
State - ST

These namespaces are stripped off when the framework queries the variable value from the indicated source. Variables derived from the GET/POST data don't have to be qualified.

For example, your page might contain:
<h2><%=AS::company%> <%=AP::product%> (<%=PG::version%>)></h2>
which would use variables from the application-set scope, the application scope, and the page corresponding to the current request respectively.

- Page Tag.

Your application must define one or more Page tags, of the following form:
<Page name="PageName" [phases=phaseSet]
                      [options=optionSet]
                      [next=nextSpec]
                      [xheads=extraHeaders]
                      [mimetype="mimetype"] />
    <PIOPath . . ./>?
    <Template . . ./>?
    <Variable . . . />*
</Page>
The phases attribute is any combination (without duplication) of the character flags:
Return processing	- r
Preamble processing	- p
Cookie processing	- c
Custom insertions	- C
Extra headers		- h
Insertion points	- i
Postamble processing	- a
If no phases attribute is provided, your page will do no custom processing, only variable substitution in the template if there is one, and you do not need to provide a PIO.

The optionSet attribute is any combination of the character flags:
Generate all after headers	- g
Generate all				- G
Initialise state at this page	- i
no caching of template			- n
No cacheing of PIO				- N
Use this page as the start page	- s
The nextSpec attribute was described under Page Sequencing Options above; it may take values as follows:
"."	- Repeat the same page,
"+"	- Go to the next page as defined in the ASD,
"-"	- Go to the previous page as defined in the ASD,
"^"	- Go back to the first page of the application,
"pageX"	- Go to the named page,
"*varX"	- Go to the page named by variable varX,
"cond? page2, page3"	- Ternary conditional, if cond true ("1", "y", "Y"), page2, else page3,
"option# page2, page3, page4"	- Jump to a page depending on the integer zero based value of variable "option".
If no next attribute is provided, DCat will jump to the Exit page specified at application-set scope, or overridden at Application scope.
The extraHeader attribute allows for shorthand representation of some common headers, or combinations of headers as follows:
Content length			- c
Kill the connection		- k
Don't cache this response	- n
These may have implications for the way DCat works beyond the simple generation of the headers. In particular, the ‘c’ xheads flag will cause DCat to buffer all output in memory so that it’s length can be determined. Other headers, cookies etc, are also buffered. Then when translation or generation is complete, the content-length header is fixed up, and the buffered output flushed to the server.

The xheads 'n' flag generates a cocktail of headers as follows:
Cache-control: no-cache
Pragma: no-cache
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Hopefully these will dissuade most proxies and browsers from caching the output.

The xheads 'k' flag simply causes generation of:
Connection: close
> Otherwise in HTTP1.1, the server will decide whether or not to keep the connection alive.

The mimetype attribute value is used in the generation of headers for the page. It defaults to "text/html".
- PIOPath tag.

The PIO tag is optional. It takes the form:
<PIOPath path="/path/to/PIO-files" />
The path provided will be used to load page implementation objects for pages in this application. It may be defined at application-set scope. A definition at application scope overrides any value at application-set scope. If no PIO tag is present, Dcat will search for templates according to the Default Directory Structure. This is a filesystem path. A path without a leading "/" will be taken as relative to the application set, e.g. path="path/to/pios" in application-set "trivia" will be taken as ".../trivia/path/to/pios".

- Template Tag.

The Template tag is optional. It takes the form:
<Template path="/path/to/templates" />
The path provided will be used to load templates for pages in this application. It may be defined at application-set scope. A definition at application scope overrides any value at application-set scope. Similarly a definition at page scope overides that at directory scope. If no Template tag is present, Dcat will search for templates according to the Default Directory Structure. This is a filesystem path. A path without a leading '/' will be taken as relative to the application set, e.g. path="path/to/template" in application-set "trivia" will be taken as ".../trivia/path/to/template", or if trivia is an alias, the corresponding path.
- Variable Tag.

The Variable tag is used to define an application-set, application, or page scope variable. It takes the form:
<Variable name="varName" value="varValue" />
Variables in these scopes live in their own namespaces. Variables at application-set scope are visible to all applications in the set. Variables at application scope are visible only to that application. Variables at page scope are visible only to that page. The namespace of variables defined in the ASD is distinct from the separate namespaces used for cookies, request variables, state, etc.
- XML Tag.

The XML tag is optional. It takes the form:
<XML path="/path/to/xml" />
It is used to specify the path to XML files representing the data to be passed between pages in the applications of a particular application set. By default, DCat will look for these in .../appsetname/xml. The XML files stored in this directory should be named for the applications they relate to – e.g. myapp1.xml. Only applications whose state attribute is set to "hxml" or "cxml" need to have such XML files. This is a filesystem path. A path without a leading '/' will be taken as relative to the application set, e.g. path='path/to/template' in application-set 'trivia' will be taken as '.../trivia/path/to/template', or if trivia is an alias, the corresponding path.

Page Templates.
- Template page syntax.

DCat page templates - .dcat files - provide some of the capabilities associated with ASP or JSP pages. They don't support embedded code, but they do allow for generation of HTML of arbitrary complexity. Because they don't allow an arbitrary mixture of script and HTML, they are usually easier to read than many ASP or JSP pages, particularly for an HTML page designer.

If your template has an entry of the familiar form:
<%=PG:varname%>
The DCat page framework will look up the value of variable varname in the ASD description of the current page (or more generally from the context) and will substitute the value it finds there for the pseudo-tag. You don't have to provide any PIO code to accomplish this.

On the other hand, where an ASP page might have:
<table>
<%
for (var i = 0; i < limit; i++) {
    response.write("...");
}
%>
</table>
a .dcat page will simply have:
<table>
<%ipname%>
</table>
where ipname is the name of an insertion point. To provide the HTML that corresponds to the insertion point name. You must provide a non-empty imlementation of the iterpretIP() method in the PIO for the page. It might be argued that his rather sparse style makes for better maintainability. At least you know exactly where to look for the code that generates this part of the HTML.

You can also choose to handle the whole substitution process - both %=% and %% - in your PIO via the 'i' flag in the page phases. In that case you will need to provide a non-empty implementation of the PIO doInsertions() method.

TODO: Extend the format to allow components!
- Run time representation of templates.

When a template is specified for a page, the framework looks for a file of name pagename ~ ".frags" in the indicated or default directory. If this file exists, the modification time of file pagename ~".dcat" is checked. It will throw an exception if this is not found. If the modification date of the latter is later than that of the .frag file, then the .dcat file is reprocessed, and a new .frag file created. Otherwise, both files are loaded into memory.

If there is no .frag file, the .dcat file is processed. This will create the .frag file.

Either of these paths results in a copy of the .dfrag file in memory, along with an array of Fragment structs:
struct Fragment
{
   byte type;	// 0 for plain text, 1 for <%=xxx%>, and 2 for <%yyy%>
   uint pos;	// offset from the start of the in-memory file
   uint len;	// length of the fragment - in types 1 and 2 this is the
				// length between % and %.
}

class MemoryTemplate
{
   this(string pageName)
   {
      string filename = pageName ~ ".dcat";
      if (exists(pageName ~ ".frags"))
      {
         d_time cr;
         d_time ac;
         d_time md;
         getTimes(pageName ~ ".frags", cr, ac, md);
         d_time fragtime = md;
         getTimes(filename, cr, ac, md);
         if (md > fragtime)
         {
            scanTemplate(filename);
            writeFrags(pageName);
         }
         else
            loadFiles(pageName);
      }
      else
      {
         scanTemplate(filename);
         writeFrags(pageName);
      }
   }

   void scanTemplate(string filename) { ... }
   void loadFiles(string pageName) { ... }

   typedef string delegate(string) sds;

   char[] instantiate(sds vp, sds ip)

   void writeFrags(string pageName) { ... }
   uint readFrags(string pageName) { ... }
}
The framework guesses at a buffer size of 3/2 times the length of the memory file, then concatenates the fragments, translating types 1 and 2 from the context data sources in the process by passing appropriate delegates to instantiate(), and reallocating on a 3/2 basis as neccessary.
- Compiled template file format.

When a template is processed it is written to a .frag file so that the scanning process does not have to be repeated unless the timestamp of the template changes. The file format consists of simple records:
  type      fragment offset 4 bytes        fragment length 4 bytes
|-------|-------|-------|-------|-------|-------|-------|-------|-------|
PIO Interface.
- The interface.

To reiterate, the interface that a PIO must implement is as follows:
interface PIO
{
   string returnProc(string prevpage, string nextpage, DCatContext* context);
   void preamble(DCatContext* context);
   string interpretIP(string ipname, DCatContext* context);
   string doInsertions(FragmentList() fl, DCatContext* context);
   string[] generateHeaders(DCatContext* context);
   string generateContent(DCatContext* context);
   string[] setCookies(DCatContext* context);
   void postamble(DCatContext* context);
}
The context data is a pointer to a struct containing references to the various entities that can provide context data:
struct DCatContext
{
   const(string[string]) requestdata;
   const(string[string]) cookies;
   const(TinyXML) state;
   const(ApplicationSet) appset;
   bool delegate(string objectName, Object object) holdObject;
   Object delegate(string objectName) retrieveObject;
   void delegate(string objectName) releaseObject;

- The returnProc method.
string returnProc(string prevpage, string nextpage, DCatContext* context);
The returnProc method is given three arguments. The first two provide the framework's view of the name of the page that submitted the request, and the page that should follow as specified by the ASD. This latter may be empty. The third argument is the context.

This method returns a string, which will generally be zero length. If an actual string is returned, the framework will take it as a 'goto' - an arbitrary page name.

This method probably has either nothing to do, or most of the important work of the application. It may have to write the results of several previous pages from a form or from the state XML to a database. Otherwise, it's probably only going to apply some rather more complex criterion for what page should follow.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The preamble method.
void preamble(DCatContext* context);
In many cases, an empty implementation will suffice here.

An important exception will be the case where the XML state representation is built progressively as the application works through pages. Such additions to the XML should probably be made at this point.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The interpretIP method.
string interpretIP(string ipname, DCatContext* context);
As noted previously, a meaningful implementation of this method must be provided if there are insertion points of the form
<% ipname >
in the corresponding page template.

It receives the name of the insertion point, and must return a string to be substituted at that point. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The doInsertions method.
string doInsertions(FragmentList() fl, DCatContext* context);
The automatic behavior of the framework for variable substitution, and the interpretIP method between tham can fully instantiate a template. The doInsertions method is included in case for some reason you wish to take full control of this process, as indicated by the 'i' flag in the page phases information.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
string[] generateHeaders(DCatContext* context);
- The generateHeaders method.
string[] generateHeaders(DCatContext* context);
This method should generate any exotic headers required by the application's next page. Note that some extra headers can be generated automatically as per the xheads options in the ASD for the page. It may well be an empty implementation.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The generateContent method.
string generateContent(DCatContext* context);
This method must have an implementation in those cases where you specify 'g' or 'G' in the page options. If you specify 'G' you must generate all the headers, including the content-length if that is required.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The generateCookies method.
string[] setCookies(DCatContext* context);
This is similar to the generateHeaders method except that you only have to generate the text for the actual cookies, each prefixed by a string such as "+20y!", "+3m!", "+10w!", "+30d!", "+12h!", or "-!" to govern its expiration. The "-!" form will put it in the past.

If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.
- The postamble method.
void postamble(DCatContext* context);
This method should do any cleaning up that was engendered by prvious PIO methods. See "Communication between PIO stages"
- PIO exceptions.

The exception class PIO Exception differs from a plain Exception in that it contains two separate messages. One of these - the regular toString() message, contains the error message that will be written to the application log file.

The other, accessible through the displayMessage() property is the one that will be used in the generation of the error page.
class PIOException : Exception
{
   string _polite;

   this(string vulgar, string polite)
   {
      super(vulgar);
	  _polite = polite;
   }

   string displayMessage() { return _polite; }
}
- Communication between PIO stages.

Three delegates are pased to PIO methods via the context information. The first
bool delegate(string objectName, Object object) holdObject;
can be used to persist information that may be required by later stages of processing. The object supplied to it will be parked somewhere by the framework. The second
Object delegate(string objectName) retrieveObject;
can retrieve such a parked object. The third
void delegate(string objectName) releaseObject;
tells the framework that you have no further need of the object and that it may thus dispose of its reference to it and leave it to the mercy of the garbage collector.

The framework will in any case, at the end of page processing, release such references.

Cacheing.
- What is cached.

The ASD for a Dcat installation is cached when the application is started, and remains in memory throughout its lifetime.

PIOs and processed templates are cached by default, but you can explicitly state than they should not be cached in the Page definition. You might want to do this for pages that are infrequently used.

Temporary objects used for communication between PIO phases are released as soon as the page processing is completed.

[TODO: Implement timing out of caching on a MRU basis controlled by an attribute on the application.]
Logging.
- Log4D.

Logging is done via module bevutils.log4d (.../dcat/bevutils/log4d.d). This also is work-in-progress, but has served well so far, being the primary debugging tool used during the construction of DCat.

There are two methods of concern in modifying or extending the DCat code
void log(int level, string msg);
void logF(A...)(int level, string fmt, A args);
Levels are:
  • ERROR: 0,
  • WARNING: 1,
  • INFO: 2,
  • VERBOSE: 3,
  • DEBUG: 4
By default DCat maintains up to 10 current and rolled over log files of maximum size 1000000. Before the request has been parsed to any extent, logging goes to .../dcat/dcat....log. After the request URI is discovered, logging switches to .../dcat/appname/appname....log.

[TODO: Add file name and line number information to Log4D in DEBUG mode. Find a way to make log() and logF() both log().]
Creating a PIO.
- Boilerplate implementation.

A generic starting point for implementation of a PIO is as follows:
module dcat.pagename;

import std.string;
import std.stdio;
import std.stream;
import bevutils.log4d;
import dcat.memtemp;
import dcat.asdreader;
import dcat.loader;

/*
struct DCatContext
{
   string[string] postdata;
   string[string] cookies;
   TinyXML state;
   ApplicationSet appset;
   Application app;
   Page page;
   bool delegate(string objectName, Object object) holdObject;
   Object delegate(string objectName) retrieveObject;
   void delegate(string objectName) releaseObject;
}
*/

class PIOImpl : dcat.loader.PIO
{
   Log4D _log;

   this(Log4D log)
   {
      _log = log;
      _log.log(4, "PIOImpl ctor done");
   }

   string returnProc(string prevpage, string nextpage, dcat.loader.DCatContext* context) { return ""; }
   void preamble(DCatContext* context) {}
   string interpretIP(string ipname, DCatContext* context) { return ""; }
   string doInsertions(MemoryTemplate mt, DCatContext* context) { return ""; }
   string[] generateHeaders(DCatContext* context) { return null; }
   string generateContent(DCatContext* context) { return ""; }
   string[] setCookies(DCatContext* context) { return null; }
   void postamble(DCatContext* context) {}
}

PIO makePIO(Log4D log)
{
   PIOImpl pio = new PIOImpl(log);
   return pio;
}
- Caveats on dynamic loading of PIOs.

At present, the dynamic loading sub-system is minimal, because minimal is likely to be quick. As such, it is quite possible that when you create a PIO and it is loaded, the load-time linker will spew out a list of unsatisfied external names. This could be dealt with programatically by having the linker load and link the module from Phobos that contained the unsatisfied external, and at some point it may be. However, it goes against my grain to have an application do some complex operation every time it is started that could be sorted out when it was built.

For the present, it is the programmers responsibility to provide fixups for such unsatisfied externals that will ensure that the build-time linker pulls them in when it creates the DCat executable. This is done by editing the module dcat.fixups (/dcat/dcat/fixups.d).

The primary trick for pulling in externals is as follows:
extern (C)
{
   extern int D9invariant12_d_invariantFC6ObjectZv;
   int* p = &D9invariant12_d_invariantFC6ObjectZv;
   extern int D3std4conv7__arrayZ;
   int* q = &D3std4conv7__arrayZ;
}
Note that the log file will report the unsatisfied external as e.g. "_D3std4conv7__arrayZ". The objects referenced are not neccessarily ints, that's just a C programmers choice of default type - all we need is an address.

If your PIO implementation uses templates, you should also define these in fixups.d, otherwise you will get a slew of unsatisfied externals.
Building and Installing DCat.
- Building.

Currently, your choices of environment are Windows, or Windows. I have a box ready to install Linux and try to port DCat to D1.x, but I haven't got around to doing that yet.

If you unzip the provided file onto your Windows D drive, then you will not have anything else to do. If you unzip it elsewhere, you will have to edit the file ...\dcat\basic.asd to reflect where you have installed it (Appset/base).

To build, open a command prompt, change directory to ...\dcat, and enter:
make -fdcat.mak
You don't need to do this initially, since the executable is included in the zip. To test it, you'll need to enter:
dcat -i
which should install the service, and then:
net start apache2
net start dcat
This of course asumes that you have Apache2 installed on your machine. For Windows, this is not difficult - see apache.org. Look at the next section of this document for configuration details.
Configuring Apache.
- Setting up httpd.conf.

Some small modifications are required to the as-installed Apache httpd.conf file. Make sure that the loaded modules section includes
LoadModule jk_module modules/mod_jk.so
Then add some sections at the end of httpd.conf so you have
JkWorkersFile d:/dcat/workers.properties
JkLogFile d:/dcat/mod_jk.log
JkLogLevel debug

<VirtualHost localhost>
ServerName localhost
JkMount /dcat/* ajp13
</VirtualHost>
Modify the JkWorkersfile and JkLogFile entries according to your installation preferences.

Testing.
- Run the provided test application.

In your browser address bar (currently only tested with Firefox), enter:
http://localhost/dcat/app1
When it doesn't work, you can email me.
What is BEV?

Brits Eye View is an Englishman's six-year personal blog about life in Arusha, Tanzania, and previously in Bangalore, Manhattan, and the Bronx. It deals with life in general, building a house, food and drink, computer programming, opinion on current affairs, 20th century history, and so on. It may give you some insight into what life is like in 'the third world', or encourage you to visit Tanzania.