What is XTC
The XML Transaction Coordinator (XTC) research project was
initiated to explore aspects of native XML data processing. With XTC
you can store, retrieve, and process large amounts of XML data under
the control of a database management system (DBMS) that is specially
tailored to the XML data model. Specifically, XTC provides an interface
for declarative XML query processing, the standardized DOM and SAX
APIs, XML index structures, fine-grained transaction isolation for
concurrent document accesses, and crash recovery. XTC is a
multi-threaded server system that runs on a host machine and accepts
multiple client connections at a time.
The XTC project consists of
- the XTC server -- a prototype of a native XML Database Management (XDBMS) System,
- the XTC clp -- a text-based command line processor,
- the XTC explain tool -- a query visualization and explain tool, and
- the XTC driver package for XML application development.
This document serves as an introduction and an overview to the XTC system.
Installation
Requirements
The entire XTC project in implemented in Java and runs on any operating system providing for a Java Virtual Machine Version 6.0. Because the Java Version 6.0 Bytecode is not compatible downwards to Java Versions 1.4.x and 1.3.x, and because we experienced some JVM errors with Java 1.5, it is an essential requirement to provide for a Java 6.0 runtime environment.
To install and run the XTCserver, you need sufficient rights to start a service that listenes on a specific port (default 24203) on your host machine. Also, make sure the port to which the XTC server will listen is available.
Download and Extract XTC
Download the latest release of the XTC project from the download section of the XTC web page at http://www.xtc-project.de. The version number of a release, for example R0.8.1, consists of the following parts:
- The major release number (R0), indicating the overall status of the project (currently a '0' for prototype).
- The minor release number (8), and
- The development number (1), where even numbers indicate stable releases and odd numbers indicate development releases.
All components of the XTC server release are compressed into a ZIP archive. Just extract the archive into an arbitrary folder of your file system. Because we currently do not use any environment variables of the operating system, it is very important that you start each component directly at the folder it is stored in.
File System Structure
By extracting the XTC server ZIP archive, the following sub-folders are created:
- clp
- lib
- driver
- doc
- explain
- lib
- resources
- samples
- dom
- sax
- xquery
- server
- cntfiles
- logfiles
- temfiles
The top level folder contains all executables of the project. These are the text-based command line processor in clp, the visual explain tool in explain, the sample programs in sample, and the server itself in server. Inside server, the XML data managed is stored in so-called container files. They reside in the cntfiles folder. The logfiles and tempfiles folders contain log and temporary files.
The driver, stored as XTCdriver.jar in the driver folder, provides for methods to administer the server and is used for application development, to access the stored XML data via the DOM and SAX interface, or to execute XQuery statements. The doc folder contains a generated Java Documentation of the driver classes and methods. This documentation is essential for application development using XTC. Because this reference is kept short, you will find further information regarding application development in the doc directory.
Finally, the samples folder consists of three sub-folders containing a collection of sample Java programs for DOM, SAX, and XQuery access to server-side managed XML data.
Installing the Server
To install the server just change into the folder server and execute the install script (note, the following examples assume a Linux environment):
./install
There are scripts for Windows and Unix environments which invoke the Java Virtual Machine with the XTCserver.jar file. The main class of the server XTCserver.class is already specified within the Manifest file in the Java archive.
The installation procedure creates various container files in the cntfiles folder and generates the meta-data required by the XTC server. The meta-data is stored in the so-called Master Document within the server and can be examined with the command line processor.
Running XTC Components
Starting and Stopping the Server
After a successful installation, the XTC server can be started resp. stopped by the following two command in the 'server' folder:
./start ./stop
For troubleshooting, you may want to get some information about server internals. You can read the logfile at server/logfiles/XTCserver.log to do so. In Unix environments, you might want to use the tail command, e.g.:
tail -f server/logfiles/XTCserver.log
The Command Line Processor
There are several ways to connect to the server and accessing the stored data. Probably the easiest way is the command line processor clp. You can invoke the clp in the clp folder with the following command:
./clp
To connect to a server running on the localhost, this command is sufficient. Currently, XTC does not provide any real user management, therefore username and password are automatically set to 'xtc'.
If you want to connect to a server running on another machine, you need to directly invoke the Java Virtual Machine and pass the necessary connection information:
java -jar XTCclp.jar <host> <port> <username> <password>
After successfully connecting to the server, clp provides you with a command promt, like this:
[ ] XTC / =>
You can now send commands to the server. The command processor is case insensitive, therefore it does not matter whether you type in HELP or help. Furthermore, some commands have abbreviations, e.g, you can also get HELP by typing a question mark ?.
Note, clp provides a 'history' function for previously entered commands.
By typing ? or HELP, you get complete list of all commands the server can understand. Please use 'HELP' to get documentation on the clp commands.
To issue an XQuery expression, you can just type it into the clp and hit return. In the appendix, you can find all supported XQuery language constructs.
The Visual Explain Tool
XTC can process XQuery statements. The internal evaluation of these statements (ie, query optimization) can be visualized by the explain tool. To start this tool, run the script in the explain folder:
./explain
If you do so, a dialog pops up in which you need to specify the connection settings (host, port, user name, and password where default values for these parameters are already provided), and, in a second tab, the path to your dot executable and the path to the xtc server directory. To visualize queries, the explain tool relies on the graphviz graph visualization program (which also provides the dot executable). You can download graphviz for Windows and Linux from: http://www.graphviz.org.
Application Development with the XTCdriver Package
The XTCdriver provides a Java API for the XTCserver which can be embedded in Java programs to develop XML-based applications (similar to the Java Database Connectivity (JDBC) for relational data sources). In your relese, you can find the Java Documentation of the XTCdriver package in the 'driver/doc' folder.
To use the XTCdriver package, make sure that you include the driver JAR file (at driver/XTCdriver.jar) in your classpath.
If you need to establish a connection from an application program through the driver package, you need to call the static XTCdriver.getConnection() method and provide the host, port, userName, and password information:
public class XTCdriver { public static XTCconnection getConnection(String host, int port, String userName, String password) throws XTCexception; }
This method returns an XTCconnection object which is the basic interface to the XTCserver. If the server is not reachable, an XTCexception is thrown.
For more information on how to use the XTCdriver package, consult the java documentation.
The XQuery Interface
XTC also provides a JDBC-like API to issue XQuery statements. To allow for a simple access to XTC's XQuery API, you first have to get a XQueryConnection object using the method getXQueryConnection
public class XTCdriver { public static XQueryConnection getXQueryConnection (String host, // the hostname of the XTC server int port, // port number of the XTC server String username, // username for authentication String password // password for authentication ) throws XTCexception }
An XQueryConnection allows you to create an Expression object using the createExpression method which corresponds to the XQuery statement you want to evaluate:
final class XQueryConnection { public Expression createExpression() throws XQueryException }
Now, as you have successfully received an Expression object, you are ready to evaluate your query using the executeQuery method:
public interface Expression { Sequence executeQuery(String query, SequenceType seqType) throws XQueryException; }
If the system successfully answered the query, a Sequence object is returned which forms a sequence of item objects. The type of each item depends on the SequenceType you have to set before evaluating the query:
public enum SequenceType { ITEMTYPE_STRING, // The result is a sequence of Items which are all of type String (each tree in the result is represented as single string) ITEMTYPE_TYPED_ITEM, // The result is a sequence of items. Note, that the items may be heterogeneous regarding their type. ITEMTYPE_DOM // The result is a sequence of DOM (root) nodes (each tree in the result forms its own DOM tree) };
The Sequence object being exposed as query result provides the following interface:
public interface Sequence extends Iterable<Item> { public Item getItem(int i) throws IndexOutOfBoundsException, XQueryException; public int count() throws XQueryException; public XMLStreamReader getAsStream() throws XQueryException; public Iterator<Item> iterator(); public void close() throws XQueryException; }
The Item interface looks like this:
public interface Item { public String getString() throws XQueryException; public Boolean getBoolean() throws XQueryException; public Integer getInt() throws XQueryException; public Short getShort() throws XQueryException; public Long getLong() throws XQueryException; public Byte getByte() throws XQueryException; public Float getFloat() throws XQueryException; public Double getDouble() throws XQueryException; public Node getNode() throws XQueryException; public BigDecimal getAsBigDecimal() throws XQueryException; public BigInteger getAsBigInteger() throws XQueryException; public void close() throws XQueryException; public ItemType getType() throws XQueryException; public boolean instanceOf(ItemType type) throws XQueryException; public String getName() throws XQueryException; public String getAsString() throws XQueryException; }
Let's have a look at at a short example using, where we query XTC's default document 'sample.xml' is queried using a simple XQuery statement. As a return type we choose ITEMTYPE_STRING, which allows as to use the Item's getAsString method:
... XQueryConnection connection = XTCdriver.getXQueryConnection("localhost", 24203, "xtc", "xtc"); Expression expr = connection.createExpression(); Sequence resultSequence = null; Sring query = "for $ b in doc('sample.xml')//bib/book " + "return " + "<name>{" + "$b/author/text()" + "}</name>"; resultSequence = expr.executeQuery(query, SequenceType.ITEMTYPE_STRING); for (Item item : resultSequence) { System.out.println(item.getAsString()); } ...
The command line output looks like this:
<name> Author1 </name> <name/> <name> Author3 </name>
Compared to the previous example, where you see an iterative style of accessing each single item of the sequence, you can also get the comlete result as an XMLStreamReader object for easy event-based processing, e.g. SAX:
... XQueryConnection connection = XTCdriver.getXQueryConnection("localhost", 24203, "xtc", "xtc"); Expression expr = connection.createExpression(); Sequence resultSequence = null; String query = "for $ b in doc('sample.xml')//bib/book " + "return " + "<name>{" + "$b/author/text()" + "}</name>"; resultSequence = expr.executeQuery(query, SequenceType.ITEMTYPE_DOM); XMLStreamReader streamReader = resultSequence.getAsStream(); XMLInputFactory inputFactory = XMLInputFactory.newInstance(); for (XMLEventReader it = inputFactory.createXMLEventReader(streamReader); it.hasNext();) { System.out.println(it.next()); } ...
For more details on using the XMLStreamReader take a look at Java 6 API Documentation.
Appendix
XQuery Support
The following grammar productions are taken from the XQuery Recommendation and summarize the XQuery language constructs implemented in XTC. Language constructs that are missing, i.e., that are not implemented or not supported, are listed below. Basically, XQuery's core features, like FLWORs, path expressions, comparison predicates, quantifications and node construction expressions are covered. Missing features are type-related expressions, module and function declarations and the if-then-else expression.
[29] EnclosedExpr ::= "{" Expr "}" [31] Expr ::= ExprSingle ("," ExprSingle)* [32] ExprSingle ::= FLWORExpr | QuantifiedExpr | OrExpr [33] FLWORExpr ::= (ForClause | LetClause)+ WhereClause? OrderByClause? "return" ExprSingle [34] ForClause ::= "for" "$" VarName PositionalVar? "in" ExprSingle ("," "$" VarName PositionalVar? "in" ExprSingle)* [35] PositionalVar ::= "at" "$" VarName [36] LetClause ::= "let" "$" VarName ":=" ExprSingle ("," "$" VarName ":=" ExprSingle)* [37] WhereClause ::= "where" ExprSingle [38] OrderByClause ::= ("order" "by") OrderSpecList [39] OrderSpecList ::= OrderSpec ("," OrderSpec)* [40] OrderSpec ::= ExprSingle OrderModifier [41] OrderModifier ::= ("ascending" | "descending")? [42] QuantifiedExpr ::= ("some" | "every") "$" VarName "in" ExprSingle ("," "$" VarName "in" ExprSingle)* "satisfies" ExprSingle [46] OrExpr ::= AndExpr ( "or" AndExpr )* [47] AndExpr ::= ComparisonExpr ( "and" ComparisonExpr )* [48] ComparisonExpr ::= RangeExpr ( (ValueComp | GeneralComp | NodeComp) RangeExpr )? [49] RangeExpr ::= AdditiveExpr ( "to" AdditiveExpr )? [50] AdditiveExpr ::= MultiplicativeExpr ( ("+" | "-") MultiplicativeExpr )* [51] MultiplicativeExpr ::= UnionExpr ( ("*" | "div" | "idiv" | "mod") UnionExpr )* [52] UnionExpr ::= IntersectExceptExpr ( ("union" | "|") IntersectExceptExpr )* [53] IntersectExceptExpr ::= UnaryExpr ( ("intersect" | "except") UnaryExpr )* [58] UnaryExpr ::= ("-" | "+")* PathExpr [60] GeneralComp ::= "=" | "!=" | "<" | "<=" | ">" | ">=" [61] ValueComp ::= "eq" | "ne" | "lt" | "le" | "gt" | "ge" [62] NodeComp ::= "is" | "<<" | ">>" [68] PathExpr ::= ("/" RelativePathExpr?) | ("//" RelativePathExpr) | RelativePathExpr [69] RelativePathExpr ::= StepExpr (("/" | "//") StepExpr)* [70] StepExpr ::= FilterExpr | AxisStep [71] AxisStep ::= (ReverseStep | ForwardStep) PredicateList [72] ForwardStep ::= (ForwardAxis NodeTest) | AbbrevForwardStep [73] ForwardAxis ::= ("child" "::") | ("descendant" "::") | ("attribute" "::") | ("self" "::") | ("descendant-or-self" "::") | ("following-sibling" "::") | ("following" "::") [74] AbbrevForwardStep ::= "@"? NodeTest [75] ReverseStep ::= (ReverseAxis NodeTest) | AbbrevReverseStep [76] ReverseAxis ::= ("parent" "::") | ("ancestor" "::") | ("preceding-sibling" "::") | ("preceding" "::") | ("ancestor-or-self" "::") [77] AbbrevReverseStep ::= ".." [78] NodeTest ::= KindTest | NameTest [79] NameTest ::= QName | Wildcard [80] Wildcard ::= "*" | (NCName ":" "*") | ("*" ":" NCName) [81] FilterExpr ::= PrimaryExpr PredicateList [82] PredicateList ::= Predicate* [83] Predicate ::= "[" Expr "]" [84] PrimaryExpr ::= Literal | VarRef | ParenthesizedExpr | ContextItemExpr | FunctionCall | Constructor [85] Literal ::= NumericLiteral | StringLiteral [86] NumericLiteral ::= IntegerLiteral | DecimalLiteral | DoubleLiteral [87] VarRef ::= "$" VarName [88] VarName ::= QName [89] ParenthesizedExpr ::= "(" Expr? ")" [90] ContextItemExpr ::= "." [93] FunctionCall ::= QName "(" (ExprSingle ("," ExprSingle)*)? ")" [94] Constructor ::= DirectConstructor | ComputedConstructor [95] DirectConstructor ::= DirElemConstructor [96] DirElemConstructor ::= "<" QName DirAttributeList ("/>" | (">" DirElemContent* "</" QName S? ">")) [97] DirAttributeList ::= (S (QName S? "=" S? DirAttributeValue)?)* [98] DirAttributeValue ::= ('"' (EscapeQuot | QuotAttrValueContent)* '"') | ("'" (EscapeApos | AposAttrValueContent)* "'") [99] QuotAttrValueContent ::= QuotAttrContentChar | CommonContent [100] AposAttrValueContent ::= AposAttrContentChar | CommonContent [101] DirElemContent ::= DirectConstructor | CommonContent | ElementContentChar [102] CommonContent ::= PredefinedEntityRef | CharRef | "{{" | "}}" | EnclosedExpr [109] ComputedConstructor::= CompDocConstructor | CompElemConstructor | CompAttrConstructor | CompTextConstructor [110] CompDocConstructor ::= "document" "{" Expr "}" [111] CompElemConstructor::= "element" (QName | ("{" Expr "}")) "{" ContentExpr? "}" [112] ContentExpr ::= Expr [113] CompAttrConstructor::= "attribute" (QName | ("{" Expr "}")) "{" Expr? "}" [114] CompTextConstructor::= "text" "{" Expr "}" [123] KindTest ::= DocumentTest | ElementTest | AttributeTest | TextTest | AnyKindTest [124] AnyKindTest ::= "node" "(" ")" [125] DocumentTest ::= "document-node" "(" ElementTest ")" [126] TextTest ::= "text" "(" ")" [129] AttributeTest ::= "attribute" "(" (AttribNameOrWildcard)? ")" [130] AttribNameOrWildcard ::= AttributeName | "*" [133] ElementTest ::= "element" "(" (ElementNameOrWildcard)? ")" [134] ElementNameOrWildcard ::= ElementName | "*" [137] AttributeName ::= QName [138] ElementName ::= QName
The following grammar productions summarize the XQuerry features that are not implemented yet. Note, non-terminals of partly implemented features may appear twice: once, in the list of implemented features, and once here. For example, the ForClause non-terminal appears in the following list again, because the TypeDeclaration feature is not supported. In some cases, the parser may parse XQuery expressions containing any of the following constructs, but the processor may not be able to evaluate them. For module-related stuff, the parser will always raise an error.
[1] Module ::= VersionDecl? (LibraryModule | MainModule) [2] VersionDecl ::= "xquery" "version" StringLiteral ("encoding" StringLiteral)? Separator [3] MainModule ::= Prolog QueryBody [4] LibraryModule ::= ModuleDecl Prolog [5] ModuleDecl ::= "module" "namespace" NCName "=" URILiteral Separator [6] Prolog ::= ((DefaultNamespaceDecl | Setter | NamespaceDecl | Import) Separator)* ((VarDecl | FunctionDecl | OptionDecl) Separator)* [7] Setter ::= BoundarySpaceDecl | DefaultCollationDecl | BaseURIDecl | ConstructionDecl | OrderingModeDecl | EmptyOrderDecl | CopyNamespacesDecl [8] Import ::= SchemaImport | ModuleImport [9] Separator ::= ";" [10] NamespaceDecl ::= "declare" "namespace" NCName "=" URILiteral [11] BoundarySpaceDecl ::= "declare" "boundary-space" ("preserve" | "strip") [12] DefaultNamespaceDecl ::= "declare" "default" ("element" | "function") "namespace" URILiteral [13] OptionDecl ::= "declare" "option" QName StringLiteral [14] OrderingModeDecl ::= "declare" "ordering" ("ordered" | "unordered") [15] EmptyOrderDecl ::= "declare" "default" "order" "empty" ("greatest" | "least") [16] CopyNamespacesDecl ::= "declare" "copy-namespaces" PreserveMode "," InheritMode [17] PreserveMode ::= "preserve" | "no-preserve" [18] InheritMode ::= "inherit" | "no-inherit" [19] DefaultCollationDecl ::= "declare" "default" "collation" URILiteral [20] BaseURIDecl ::= "declare" "base-uri" URILiteral [21] SchemaImport ::= "import" "schema" SchemaPrefix? URILiteral ("at" URILiteral ("," URILiteral)*)? [22] SchemaPrefix ::= ("namespace" NCName "=") | ("default" "element" "namespace") [23] ModuleImport ::= "import" "module" ("namespace" NCName "=")? URILiteral ("at" URILiteral ("," URILiteral)*)? [24] VarDecl ::= "declare" "variable" "$" QName TypeDeclaration? ((":=" ExprSingle) | "external") [25] ConstructionDecl ::= "declare" "construction" ("strip" | "preserve") [26] FunctionDecl ::= "declare" "function" QName "(" ParamList? ")" ("as" SequenceType)? (EnclosedExpr | "external") [27] ParamList ::= Param ("," Param)* [28] Param ::= "$" QName TypeDeclaration? [30] QueryBody ::= Expr [32] ExprSingle ::= TypwswitchExpr | IfExpr // for clause *with* type declaration [34] ForClause ::= "for" "$" VarName TypeDeclaration PositionalVar? "in" ExprSingle ("," "$" VarName TypeDeclaration PositionalVar? "in" ExprSingle)* // let clause *with* type declaration [36] LetClause ::= "let" "$" VarName TypeDeclaration ":=" ExprSingle ("," "$" VarName TypeDeclaration ":=" ExprSingle)* [38] OrderByClause ::= ("stable" "order" "by") OrderSpecList // no empty least or empty greatest and no collations [41] OrderModifier ::= ("ascending" | "descending")? ("empty" ("greatest" | "least")) ("collation" URILiteral) // no quantification *with* type declaration [42] QuantifiedExpr ::= ("some" | "every") "$" VarName TypeDeclaration "in" ExprSingle ("," "$" VarName TypeDeclaration "in" ExprSingle)* "satisfies" ExprSingle [43] TypeswitchExpr ::= "typeswitch" "(" Expr ")" CaseClause+ "default" ("$" VarName)? "return" ExprSingle [44] CaseClause ::= "case" ("$" VarName "as")? SequenceType "return" ExprSingle [45] IfExpr ::= "if" "(" Expr ")" "then" ExprSingle "else" ExprSingle [54] InstanceofExpr ::= TreatExpr ( "instance"; "of" SequenceType )? [55] TreatExpr ::= CastableExpr ( "treat" "as" SequenceType )? [56] CastableExpr ::= CastExpr ( "castable" "as" SingleType )? [57] CastExpr ::= UnaryExpr ( "cast" "as" SingleType )? [59] ValueExpr ::= ValidateExpr | PathExpr | ExtensionExpr [63] ValidateExpr ::= "validate" ValidationMode? "{" Expr "}" [64] ValidationMode ::= "lax" | "strict" [65] ExtensionExpr ::= Pragma+ "{" Expr? "}" [66] Pragma ::= "(#" S? QName (S PragmaContents)? "#)" [67] PragmaContents ::= (Char* - (Char* '#)' Char*)) [84] PrimaryExpr ::= OrderedExpr | UnorderedExpr [95] DirectConstructor ::= DirCommentConstructor | DirPIConstructor [101] DirElemContent ::= CDataSection [103] DirCommentConstructor ::= "<!--" DirCommentContents "-->" [104] DirCommentContents ::= ((Char - '-') | ('-' (Char - '-')))* [105] DirPIConstructor ::= "<?" PITarget (S DirPIContents)? "?>" [106] DirPIContents ::= (Char* - (Char* '?>' Char*)) [107] CDataSection ::= "<![CDATA[" CDataSectionContents "]]>" [108] CDataSectionContents ::= (Char* - (Char* ']]>' Char*)) [109] ComputedConstructor::= CompCommentConstructor | CompPIConstructor [115] CompCommentConstructor ::= "comment" "{" Expr "}" [116] CompPIConstructor ::= "processing-instruction" (NCName | ("{" Expr "}")) "{" Expr? "}" [117] SingleType ::= AtomicType "?"? [118] TypeDeclaration ::= "as" SequenceType [119] SequenceType ::= ("empty-sequence" "(" ")") | (ItemType OccurrenceIndicator?) [120] OccurrenceIndicator::= "?" | "*" | "+" /* xgs: occurrence-indicators */ [121] ItemType ::= KindTest | ("item"; "(" ")") | AtomicType [122] AtomicType ::= QName [123] KindTest ::= SchemaElementTest | SchemaAttributeTest | PITest | CommentTest [125] DocumentTest ::= "document-node" "(" SchemaElementTest ")" [127] CommentTest ::= "comment" "(" ")" [128] PITest ::= "processing-instruction" "(" (NCName | StringLiteral)? ")" [129] AttributeTest ::= "attribute" "(" (AttribNameOrWildcard ("," TypeName)) ")" [131] SchemaAttributeTest ::= "schema-attribute" "(" AttributeDeclaration ")" [132] AttributeDeclaration::= AttributeName [133] ElementTest ::= "element" "(" (ElementNameOrWildcard ("," TypeName "?"?)?)? ")" [135] SchemaElementTest ::= "schema-element" "(" ElementDeclaration ")" [136] ElementDeclaration ::= ElementName [139] TypeName ::= QName [140] URILiteral ::= StringLiteral