Content Assembly Mechanism (CAM) News

The new CAM v2.0 now available from Sourceforge

2011-07-09T12:20:00.000-07:00

Work continues on improving the CAM editor toolkit - and the latest release boasts an impressive feature set including:

Drag and drop assembly of exchange structures from sets of dictionary canonical components
Support for modelling with UML/XMI and “FreeMind” MindMap tools
Cross platform deployment - Windows, *NIX and MacOs in 32bit and 64bit executables
New visual editor with color coding, 3x performance increase and 50% less memory use
Enhanced and improved NIEM IEPD production features and NIEM dictionary collections
Dictionary harvesting from schema with dictionary compiling for drag and drop tool use
External codelist generation and support
Support for branding, delivery packaging and NIEM features development from Oracle
Standalone CAMV 1.1.4 validation engine and ANT batch scripts testing environment

Together these improvements allow people to fully integrate their exchange schema development from
dictionaries of components to exchange schema structures. Then align with modelling documentation
and designs, while automating the production of actual exchange schema and XML instance test suites.

The full release notes PDF are here.

You can find out more from the CAM editor download site.

Latest CAMeditor v1.9 now available from Sourceforge

2011-02-02T19:50:00.000-08:00

The latest CAMeditor release is now available from Sourceforge:

http://sourceforge.net/projects/camprocessor/

Highlights include:

1) Complete external code lists support for code values via files and import of code lists

2) Enhancements to the evaluator, NDR checks and NIEM re-use scoring tools

3) Performance enhancements to XSD schema importing tool

4) Generation of NIEM EIEC / BIEC dictionary schema from ERwin enterprise model components

5) Support for KeyRef notations in structure editor

6) Enhanced XML example generation

Then there's a raft of fixin's and improvements from over 4 months of development since the last release. Full details are in the release notes PDF documentation and the online tracker system.

Thanks to everyone who has contributed to helping make this new release the best yet. And special shout out goes to IEPD Factory team and working with us so we can share code list files between tools.

Please feel free to provide feedback via Sourceforge project discussion area, OASIS CAM dev list, or contacting the developer team directly.

Who needs OASIS any more?

2010-06-23T07:35:00.000-07:00

Is the day rapidly approaching where we have essentially made ourselves redundant?

I gave testimony today in New York on their proposed legislation on open government and open data. This whole world of Web 3.0 collaborative information and Gov 2.0 solutions is supposed to be self-describing as per the W3C and their “open data” RDF syntax.

More telling is that the community is driven by “give us the data” and they care little for how they get there; spreadsheets, csv, text, XML, or RDF. So injecting a structured approach into the conversation is difficult given the myriad of agendas being advanced and the timelines desired. Only once the data Tower of Babel becomes intractable under its own weight will some perspective of the long term challenges become apparent? So perhaps a parallel path is possible, where these data providers go off and get the data and meanwhile someone else begins the process of cataloguing the data entities and content models involved and looking to move toward an aligned future view. This could be the opportunity for OASIS to develop “CityHallML” providing the most common components of financial, infrastructure and performance data along with census and demographic data.

The city of course is looking at the costs of getting data and then supporting that with archives and updates and publication feeds. Having a harmonized approach can potentially significantly reduce deployment and sustainment costs along with potential software development collaboration and cost savings for cities themselves. Having a common view also of course helps solution providers market to cities nationally not just locally. Perhaps the biggest challenge is the unspoken one of complexity. The more one steps into data sharing one sees the opportunity for people to interpose complexity. Keeping things simple, yet consistent and transparent requires constant vigilance and oversight to ensure that solution providers are not injecting their own self-serving complexity. After all complexity costs money to build and support, is a barrier to competitors, and hence vendors are naturally drawn to inject complexity.

Today I also published a paper on creating dictionaries of canonical components, aligned to the NIEM.gov approach and CEFACT core components model. This would be a great foundation for CityHallML.

http://www.oasis-open.org/committees/document.php?document_id=38385

This juxtaposes with the W3C world view of self-describing data instances and RDF. You have the approach of either the embedded RDF semantics, with all that overhead on each and every data item, or you have this OASIS-based approach of semantics referenced in domain dictionaries and information structure templates that allow comparatively small concise data instances where the XML tags provide the content referencing between content and semantics about the content.

Equally important is that the canonical components are built using naming and design rules (NDR) that drive consistency of approach and convergence on terms and meaning.

This all contrasts with today’s approach of publishing mega-structure as a Schema that contains all possible exchange components for every facet of a business process. This then forces developers to unravel the puzzle of what each part of the business process needs from that mega-structure, often sending redundant or empty data elements, instead of dynamic content assembly templates using selected parts from a dictionary of canonical components.

Now, lets assume everyone drinks this OASIS Cool-Aid - they create domain dictionaries of canonical components, and then use shared open source tooling to create their information structures dynamically and the tooling takes care of all the plumbing, templates, extraction and creation of XML instances from backend data stores, and submission to XML online repositories for archiving and exposure through search and retrieval services.

Vendors and government collaborate to develop and deploy open source based portals that allow further sharing and open access to data. Additional niche services using collaborative social platform tools integrate into these and deliver a wealth of community facing solutions to citizens.

Life is good.

This means technically no one strictly needs to publish formal exchange structure schema any more, exchanges are dynamically built to purpose by the communities. We already saw this need happen recently during the Haiti relief effort, when OASIS Emergency EDXL had to be extended on the fly to support on the ground situations with hospitals and the services they can provide.

So then who needs OASIS any more at this point? How do we make ourselves relevant in this uber Web 3.0 world and data sharing dominated by XML based services driven by today’s technology underpinnings of SOAP, REST, RSS and http, IETF and W3C speak with RDF?

Where is the long term picture for OASIS in all this? And is this necessarily a bad thing? We could just decide that "mission accomplished" is the good answer.

After all the recent demise of "shareware" shows that the world has got the message in marketing terms of building solutions that are open source and publish and use open publicly released standards. Then these are managed with peer review within a user community and a stakeholder authoritative source providing stewardship. Perhaps providing collaboration support services to such communities could be another OASIS facility.

We could envision that there would be the need for a triumvirate to manage and steward the go forward where federal, state and local government stakeholders need independent oversight and technology guidance. This is similar to what NIEM.gov is currently doing federally and then perhaps as New York and other states would seek for.

Of course many vendors are out there pitching their wares and setting up stall, figuring if they can own a states data then they essentially have a license to print money from those needing access to data or pushing targeted advertizing content at them along with the data they seek. In New York was heard testimony that “It’s only a small monthly fee or one time subscription for a week’s access to what you need and we have analysts to help you”. Notice also that Microsoft has created OData to publish RSS driven feeds that link also into SharePoint, and then Google has its own open data APIs available and associated search tools.

So for OASIS this could be positioning in terms of long term objectives keeping data sharing truly open without the dominance of particular solution providers at the expense of smaller community based services, or even the community itself. Information empowers democracy but can also be used to track and restrict freedoms of those who would seek that truth and equality. Asking suspicious questions can incur penalties or allow law enforcement to track potential suspects.

Plus the traditional areas of formal legislated transactional information exchanges for secure B2B use may provide OASIS with continued niche work. Although the gap there will continue to blur as the use case for open data encroaches on transactional data and network speeds continue to erode what is thought of as optimized high volume exchanges with small transactions.

Then that is also blurred by building contextual business process driven data exchanges from components drawn from canonical dictionary collections with embedded links to open data sources; e.g. I send the city a price quote for items and embed reference links to my public company profile, my digital certificate public key registered with the city, and the links and references to item descriptions published by the city for the RFP. The city itself then on contract award can simply publish that same information as was submitted as the bid.

This ushers in a very collaborative new world.

So perhaps another answer is augmenting the OASIS Kavi with Web 3.0 enabled portals and services that can publish canonical dictionaries of component definitions to help drive standardization out there in the domain user communities? Thereby keeping OASIS central to conversations around authoritative sources for good high quality components for use in building collaborative spaces?

Then just maybe the answer lies beyond data and into rule sharing and systems.

If we have solved information sharing then the next piece of the puzzle is open sharing of the under laying rules and trap doors that can snag the unwary. Clearly rule sharing systems are the next step up from just data sharing because they have to be built on top of consistent information representations.

Back in the day in 1998 when we started the XML/edi work we talked about "The Fusion of Five" - XML, EDI, repositories, templates and agents.

Each of these represents:

XML - web foundation
EDI - business methods
Repositories - reference component dictionaries
Templates - process logic for exchanges
Agents - implementation control and intelligent automation tools

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.3149&rep=rep1&type=pdf

Checking off the first 4 here as becoming closed chapters in the brave new Web 3.0 world, so is the agent piece the next great frontier? We are already seeing related work such as the OASIS SET TC that is providing a framework for information mapping automation.

Clearly the world is redefining what is perceived as possible and what requires better solutions and standard representations.

Open Government Open Data - Testimony to New York City hearing

2010-06-23T07:26:00.000-07:00

In the context of open data - Web page data is what people traditionally think of - but that is unstructured data - it is not predictable. It is typically served via portals. This makes it difficult for search engines to harvest and index and link and hence harder to find.

Structured data is used instead to publish data sets that can be freely accessed via a data mine. As data is added to the data mine however, it becomes important to align to information vocabulary standards so that information can be catalogued and discovered consistently.

Not all data is created equally. Without a vocabulary, lexicon and approach you end up with a lot of noise in the system that actually inhibits access to data as the mine grows.

Classification by cost to implement and value to the public casts insights on what the underlying challenges are. Making it possible to make data much more readily available with lower costs to not only deploy the data but then update periodically and archive older datasets is therefore technically the challenge.

Future proofing this data is also another lesson learned as technology moves extremely rapidly as we know.

What this teaches is a paradox. We want to do data right and standardize how that is done on the one hand; but we do not want that prescriptive so it blocks out new technology or innovations. We all know the risks of vendor or community lock-in so that only a limited set of providers can play or you need their special software. Using open public standards and open source friendly technology are therefore key needs.

Coupled to this is how the information can be accessed. Again, avoiding proprietary access methods is vital and using open public services that the internet infrastructure can readily access. So rather than adopting “open” vendor APIs which may be initially alluring, being careful to ensure those are not then the sole sources of data is equally important. Other public sources of directly accessing the complete raw data are a key need. Notice also that vendors APIs are two way, so they can harvest who is requesting what types of information and that may or may not be a concern for the public.

Perhaps the biggest challenge is the unspoken one of complexity. The more one steps into data sharing one sees the opportunity for people to interpose complexity. Keeping things simple, yet consistent and transparent requires constant vigilance and oversight to ensure that solution providers are not injecting their own self-serving complexity. After all complexity costs money to build and support and hence vendors are naturally drawn to inject complexity.

What this teaches is that New York could usefully invest in the development of open source software resources, and tapping into existing open source projects to leverage what is there today and act as a lead for development of data. By focusing resources on this aspect it facilitates programming and development both internally and externally. Rather than paying over and over for the same vendors developing proprietary solutions for the city.

One area that is instructive is the work of the National Information Exchange Model (NIEM.gov) and the challenges and opportunities this illustrates for federal government.

Providing a forum where government and industry can work together, so that government is not handing over the keys to data to a select few vendor companies.

Similarly reducing the cost of interoperable sharing of information exchanges by providing pre-built common vocabularies for domains that establish a common reference point is vital. This allows agencies to map their local data to the common lexicon.

This universal standard for federal government can provide a blueprint for state and local use. What is more, open source tools are available that help accelerate development of NIEM-based applications.

What NIEM is built on is the use of XML data representation techniques and open public standards for modelling and presenting data. Following these guidelines is essential to having consistent information. Allied to this using software tools to check the integrity and consistency of the data is important, because as data grows it becomes increasingly difficult to manage and check this by hand. Currently the City has published 200 datasets. How consistent are these today? Since the city is publishing data in different formats, databases, spreadsheets, text and XML it is likely it is not very consistent at all. By the way it should be mentioned it is easy to create XML data that is also spreadsheet compatible. Not many people know that nugget and this shows some of the technical challenges and knowledge sharing needed.

Last but not least an enlightening example is the case of the State of California election real time results reporting data that is broadcast on election day; this took a lot to get the State to do in terms of selecting the right technology, getting the data into an open OASIS XML public election data reporting format, and then signing up media channels to consume and report this information to the public. The past three years news channels across the state have quietly integrated this into their live website feeds.

Notice the challenges this shows for legislators and implementers. Who is the authoritative source, and how does the public verify that data they are being shown is accreditable and trustable? How do you select or foster the development of proper public standards for use with critical information feeds? How do you facilitate good aspects of data sharing such as auditing and verification of data that is being provided, including being able to have consistent data that can make historical trends.

By the way it should be noted that California found it technically easy to dump their SQL database election data directly into the OASIS XML reporting format. These types of simple script approaches would be open techniques that New York could learn from and leverage.

So what California has done on the one hand is good – making media sources the custodians, using internal election standards for presenting the information – but on the other hand is weak in that they are restricting public access to the original source and an archive of such data. Having procedures and methods in place so that the public can request access to information so that there is control over accessing more sensitive types of data, and understanding who may serve as public sources of trust is important to making more data available.

So in summary, using open public standards and resources that use open source software and leveraging open internet communication and information access methods is crucial and not slipping into a situation where a limited set of industry vendors then essentially own your data with increasingly levels of complexity to accessing that data.

XML Validation Framework using OASIS CAM (CAMV)

2010-05-13T18:50:00.000-07:00

This article on IBM DeveloperWorks explores the use of the CAMV Java engine to implement a flexible XML validation framework.

The approach for XML Validation using OASIS Content Assembly Mechanism (CAM) templates supports a wide array of complex message exchanges with business partners using B2B or B2C business patterns. The CAM templates simplify and externalize the validation rules while allowing the gateway to act as a pass-through on information that is not directly relevant.

Read the full article at: http://www.ibm.com/developerworks/library/x-camval/index.html

NIEM Tools example - NIEM and LEXS dictionary spreadsheet

2010-03-08T10:45:00.001-08:00

Continuing NIEM Tools support in CAM now includes a dictionary that contains NIEM 2.0, NIEM 2.1 and LEXS 3.1.4 components. This is available in a handy Excel spreadsheet format as a NIEM tool for reuse analysis and mapping determination.

The LEXS dictionary also works with the latest CAM blueprints and expander tool - to assemble exchange schema structures directly from the dictionary definitions into resulting CAM templates.

More information on CAM blueprints is available from the CAM documents area on OASIS.

For more information on NIEM Tools see also the NIEM Tools link on wikipedia.

NIEM tools wikipedia and CAM toolkit

2010-02-27T16:38:00.000-08:00

The Wikipedia entry on NIEM (National Information Exchange Model) now includes a catalog of third party IEPD (Information Exchange Package Documentation) tools which features information on the CAM toolkit and its support for NIEM IEPD development.

For more information see the Third Party NIEM tools section on wikipedia.

New CAMeditor v1.7 with NIEM 2.1 and LEXS 3.1.4 support

2009-12-11T20:28:00.001-08:00

This release completes the series of 2009 releases and consolidates 3 months of development work on both CAMV validator engine and the CAMeditor tools.

For the CAMeditor significant changes include improvements to the Eclipse user interface and template structure display along with enhancing the top down designer and generation and handling of large dictionary structures. Also included is a new LEXS 3.1.4 dictionary with sample expander blueprint templates for LEXS messages and updates of the NIEM dictionary files to the NIEM 2.1 release. Various NIEM related enhancements have been made in support of better IEPD generation ( http://www.niem.gov ). CAMeditor is built using Eclipse, Java, and Saxon xslt.

The CAMV validation engine is now a thread-safe implementation supporting deployment in middleware containers such as jBOSS or IBM Websphere MQ™. Validation of exchange structures now allows handling of very large XML instances with checking of a discreet subset of business content requirements. Also integration support for Java call methods (SDOM) has been implemented (CAMV is developed in Java using Saxon, Xerces and XPath v2.0 support).

The project vision is to provide the leading open source toolset for implementing standards based information exchanges with XML, including the NIEM IEPD approach. Simplifying and speeding the development process and enhancing the quality of your resulting schema for superior XML exchanges. To date we have had over 15,500 downloads from Sourceforge.net (http://www.sourceforge.net/projects/camprocessor ).

Here’s wishing everyone a Very Happy Holiday Season and all success in 2010 for your XML projects.

New JCAM v1.6.9 with XML dictionary based exchange top down designer

2009-10-10T17:33:00.000-07:00

The new release of CAM Editor toolkit is now available:

http://sourceforge.net/projects/camprocessor/

Now includes dictionary driven core components for use in top down exchange design.

For a live action demonstration that illustrates this in action at the recent NIEM Expo' see (signup is free):

http://tinyurl.com/cam-niem

In a nutshell if the template expander tool matches a node in your outline structure with a component in the dictionary - then it will insert the complete structure fragment, nodes, rules and annotations into your template.

The expand tool uses a repository configuration file you set to point to the dictionaries to use (sample also in dictionary directory).

You can build dictionaries from any sets of existing schema - the OASIS EML v6 dictionary is included in the examples provided.

And as usual raft of bug fixes and enhancements to existing features - please see the release notes PDF for specifics.

Release 1.6.7 July 2009 is now available

2009-07-07T14:40:00.000-07:00

The new release of camprocessor is available on SourceForge:
http://www.sourceforge.net/projects/camprocessor

The NIEM wantlist online SSGT upload is now supported allowing simple generation of a wantlist by comparing to the NIEM dictionary.

Selection of bug fixes and enhancements from June update cycle.

Comparing Two XSD schema made easy

2009-05-30T10:23:00.000-07:00

One of the benefits of the visual approach to XML in CAM templates is that it makes it easy to compare two schema.

By ingesting one or more XSD schema into CAM templates - it is then easy to use a standard compare tool (such as KDiff open source) - to review the changes and differences in the structure and rules separately.

By contrast the complexity of XSD schema syntax makes direct comparison of two XSD schema problematic - especially if the changes are buried in import or include files.

Using the CAM approach solves that by making templates that can be simply compared. Of this works for CAM templates themselves - and this was a design intent - to make it easy to review changes between versions of partners exchange schema templates.

CAM open source tool available from: http://www.sourceforge.net/projects/camprocessor

New release 1.6.5 of OASIS CAM toolkit for exchange schema

2009-05-30T10:22:00.000-07:00

The new release of camprocessor is available on SourceForge:
http://www.sourceforge.net/projects/camprocessor

New features include enhanced tool for evaluating a schema and template against naming and design rules (NDR). The evaluator is capable of flagging a whole range of common mistakes and issues from XSD schema.

The term representation rules and the assessment score are all configurable via xslt functions that are in a separate include file.

Also improved is the XSD schema ingesting process to cover more syntax nuances of XSD.

Then the CAM 2 XSD exporter is more comprehensive in its generation of XSD schema - including support for NIEM.gov features and wantlist generation by comparison to the NIEM dictionary.

The XML test case samples generation has also been improved. Overall 38 enhancements and fixes added.

DEVX articles on using jCAM information exchanges

2009-04-26T12:49:00.000-07:00

A series of articles by Michael Sorens for DEVX magazine provide in depth insights into the OASIS CAM technology and tutorial details and examples.

Michael's 3 articles can be found here:

Taking XML Validation to the Next Level: Introducing CAM

Taking XML Validation to the Next Level: XSD Schema vs. CAM

Taking XML Validation to the Next Level: Explore CAM's Expressive Power

jCAM processor hits 5,000 download landmark and 20 downloads

2008-10-11T11:40:00.000-07:00

I'm very proud to announce that the jCAM implementation of OASIS CAM v1.1 available on SourceForge has passed 5,000 downloads and this week has been averaging 20 downloads a day.

http://sourceforge.net/project/stats/?group_id=97941&ugn=camprocessor

This is a huge tribute to the dedication and hard work of Martin Roberts particularly in building the Eclipse based jCAM implementation.We have also added a suite of XSLT tools to the jCAM package that provide ability to ingest and manipulate XSD schemas along with producing automated documentation and XML test case instances.

These have enabled us to work with a selection of industry standard schemas enhancing and crosschecking and validating the XSD schema work.

The jump start tutorial is available here:
http://www.oasis-open.org/committees/download.php/29381/EML%20and%20jCAM%20tutorial.pdf

And the project resource site is: http://www.jcam.org.uk/

Updated White Paper on CAM v1.1 and W3C Schema v1.1 Insights

2008-09-20T13:00:00.000-07:00

Updated White Paper on CAM v1.1 and W3C Schema v1.1 Insights

FYI, after review edits and comments:http://www.oasis-open.org/committees/download.php/29164/White%20Paper%20on%20CAM%20and%20XSD.pdf

Thanks, DW
---------------------------------------------------------------------
To unsubscribe, e-mail: cam-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: cam-dev-help@lists.oasis-open.org

White Paper on CAM v1.1 and W3C Schema v1.1 Insights

2008-09-06T07:31:00.000-07:00

This white paper has been produced to discuss using CAM technology with schema.

You access the white paper here.

freeb-UBL site launched using CAM templates

2007-02-01T10:38:00.000-08:00

For more details see: http://xml.coverpages.org/freeb-ubl-Announce.html

New release of jCAM available

2006-01-10T07:56:00.000-08:00

The latest release is now available from http://camprocessor.sourgeforge.net

OASIS CAM TC

2005-08-10T09:17:00.000-07:00

Initial posting - for information on CAM specifications see:

http://www.oasis-open.org/committees/cam