Foswiki to MDM Static Site migration

CitizensOversight (2026-06-18) RaymondLutz

This Page: copswiki.org/Common/FoswikiToMdmMigration

More Info: CopsWiki

Why Citizens' Oversight Moved from Foswiki to a Static Site Architecture

Citizens' Oversight has used wiki-based content management systems for more than two decades.

The original site was built using TWiki, one of the earliest wiki systems. TWiki was first released in 1998, before Wikipedia and before many of the content management systems that later became popular. At the time, the primary goal was to create a system that would allow documents, media, research, and public-interest information to be published quickly and organized effectively.

An important concern from the beginning was avoiding dependence on any particular media hosting provider. Over the years, many online media services have appeared and disappeared. Early video hosting platforms such as Google Video and others were eventually phased out, forcing content creators to migrate their work. To address this problem, Citizens' Oversight used the wiki as a persistent indexing system in which the topic describing a resource remained stable even if the underlying media moved to a different hosting service.

In 2008, the open-source community created Foswiki as a fork of TWiki. Citizens' Oversight eventually migrated to Foswiki and continued to expand the site. Over the years, thousands of topics, attachments, media resources, projects, investigations, and election-related records were added to the system.

Foswiki proved to be a mature and capable platform. Its forms, attachments, metadata, linking system, and macro language made it possible to organize a large body of information without relying on a conventional database. However, the architecture also had limitations.

Because pages were generated dynamically, every page request required server-side processing. As the site grew and more sophisticated searches and reporting features were added, some pages became increasingly expensive to generate. Certain pages that relied heavily on SEARCH macros could require many seconds to render.

At the same time, internet traffic patterns changed. Automated crawlers, scraping bots, and denial-of-service attacks became more common. In recent years, large-scale AI scraping activity has generated substantial load on many public websites. Since Foswiki generates pages dynamically, even routine crawling activity can consume significant server resources. The result was increasing vulnerability to slowdowns and service interruptions.

Upgrading Foswiki also became increasingly difficult. While the software itself remained functional, major version upgrades often required substantial effort. It became difficult to justify another upgrade cycle when the underlying dynamic architecture remained vulnerable to the same performance limitations.

For these reasons, Citizens' Oversight began migrating to a static-site architecture.

The new system preserves the content, metadata, attachments, and organizational structure of the original wiki while eliminating the need to generate pages dynamically on the public server. Pages are generated in advance and deployed as ordinary HTML files.

The current publishing pipeline consists of several stages:

1. foswiki_to_mdm.py -- Converts the original Foswiki corpus into Markdown while preserving Foswiki macros. The resulting files use the .mdm format ("Markdown with Macros"). At this stage, topics remain editable using standard Markdown tools such as Typora. This conversion is used rarely, and repeated only as needed to refine the conversion. Core to this conversion is Pandoc, but it has a number of bugs and limitations that the code had to work around.
2. mdm_to_md.py -- Expands macros, resolves links, converts WikiWords, and produces ordinary Markdown.
3. md_to_html.py -- Converts Markdown into static HTML pages. This phase was originally started with eleventy but we found it had limitations with regard to generating internal links, and so python markdown was used.
4. Deployment -- A deployment script regenerates only the files that have changed and updates the public website.

The new system also includes command-line tools for topic creation, editing, attachment management, and metadata maintenance.

Metadata originally stored in Foswiki META records is preserved as Markdown frontmatter. Many traditional Foswiki features are implemented using Python rather than Foswiki's SEARCH macro architecture. Custom macros are evaluated during the build process using in-memory data structures built with the Daffodil package, allowing complex relationships to be maintained without requiring dynamic page generation.

The result is a site that is substantially faster, more resistant to automated traffic, easier to deploy, and less dependent on any particular hosting provider, while preserving the accumulated content and institutional knowledge developed over many years.

In many ways, the migration continues the original design goal: preserving access to information while avoiding dependence on any single technology platform.

Status and Outlook

Generally, this transition is on a successful trajectory, with the following comments:

• The site is now very responsive and should be immune to bot strikes and heavy usage.
• The editing and topic creation cycle has been streamlined and can be further improved.
• Data Forms have been improved to include a "Default" column so defaults can be provided in the form definition.
• Most uses of SEARCH have been replaced with searches over a Daffodil array using python.
• Macro Evaluation now is performed prior to deployment.
• Search box and Jump box do not exist. Instead, users can use full lists of media and search it in the browser.
• Many sections of the site will not be migrated.
• CHANGE: Evaluation of macros in INCLUDED topics is done in the context of the INCLUDING topic. This is the opposite of the method used Foswiki, but is more useful in the Copswiki situation.
• Most Macros have been implemented or have been designed out, using python based searches, for example. Here are some macros that are probably not going be implemented:
• SEARCH -- will not be implemented directly because it mixed both searching and formatting. This is instead implemented as searches over a data table of other topics and then the result formatted. Paging is not implemented, but the entire list placed on a single page. As a result, the site has no delay even for long search results.
• QUERY -- generally not needed and can be replaced by access to context.
• CALC -- calculations in spreadsheets and elsewhere.
• CALENDAR -- Not planned to be implemented.
• URLPARAM, FORMELEMENT, STARTFORM, ENDFORM -- used for interactivity.
• STARTMARKDOWN, ENDMARKDOWN -- Markdown is now standard.
• STARTPUBLISH, ENDPUBLISH -- static publishing is normal.
• SLIDESHOWSTART, SLIDESHOWEND, SLIDENUM, SLIDENAVNEXT, SLIDENAVALL, SLIDEMAX, SLIDETEXT, SLIDETITLE
• SENDEMAIL, EDITTABLE -- imply interactivity.
• SCRIPTURLPATH, SCRIPTSUFFIX, WIKIPREFTOPIC -- Not appropriate for static site. There is no script.
• IF not implemented but IFFIELD was implemented instead as IF was always used that way.
• SERVERTIME -- If needed, must be part of the JS template.
• The following may be supported:
• TABLE,
• IMAGEGALLERY
• FOOTNOTE
• INCLUDE of arbitrary url.