Datasets

The first step in data virtualization is defining where your source data is located and how it can be read. You do this by creating a dataset and filling it with one or more data sources. Data sources can be XLSX, JSON, XML, CSV or RDF files, as well as connections to SQL databases or remote APIs, and many others.

Data sources are configured automatically. If the default settings do not apply to your data source, you can adjust them manually. See the settings for different data types:

• CSV

Sparque Desk converts all data sources into virtual XML fragments. For example, given a CSV file actors.csv:

actorID,actorName
1,John Travolta
2,Uma Thurman

Sparque Desk produces the following two XML fragments:

<row>
  <field column="0" name="actorID">1</field>
  <field column="1" name="actorName">John Travolta</field>
</row>

<row>
  <field column="0" name="actorID">2</field>
  <field column="1" name="actorName">Uma Thurman</field>
</row>

The virtual XML unifies the syntax for the next step in the data virtualization process, mapping the data to a knowledge graph. The virtual XML can be more verbose than the source data. This doesn't really matter as the fragment only exists virtually. It will never be written to disk during the indexing process.

To improve the speed of uploading your data into pipelines and reloading graphs, we recommend that you upload compressed data, also known as connected pipelines. Connected pipeline compression is supported for XML, JSON, and CSV pipelines. Decompression is supported for the following formats: GZip, Z-Standard, BZip2, and Z. This section explains how the system determines which decompression method to use.

The second step in data virtualization is the mapping of the virtual XML fragments to the knowledge graph. This is visualised in the image below:

Data Mapping

For each data source, we define a mapping in the Sparque Mapping Language. The mapping language is an extension of XSLT, a transformation language for XML. As in any XSLT, you define templates that match with elements in your XML data. In this case, the virtual XML that comes out of a data source. The key differences between a normal XSLT file and a Sparque data mapping is that we do not write the normal textual output of the XSLT, but instead generate triples.

We assume that you have basic knowledge of writing XSLT. You do not need to know how to execute XSLT. Sparque takes care of the whole pipeline for you. The documentation of the mapping language explains in detail how to create a mapping:

• Mapping language

By including the su (Sparque Utilities) namespace in your data mapping, you get access to some additional utility functions. Examples are methods to transform strings, such as changing the case, splitting, and normalizing. There are also methods for processing numbers, dates and person names:

• Utilities

You have to decide yourself which attributes, relations and types (classes) you want to model and how you want to name them. This does not mean that you have to invent all of them yourself. There are many vocabularies available. The Open Knowledge Foundation provides an extensive overview of vocabularies.

We advise to reuse existing vocabularies whenever available. When your domain requires a more specific vocabulary, you can extend existing vocabularies. But in some cases, it also makes sense to invent your own vocabulary.

The third and final step in data virtualization is storing the triples that are generated by your data mapping(s) in Sparque's graph database. In this step, the data mappings that you wrote are applied to all the data in your data sources. We often refer to this step as indexing or loading.

With the completion of this step, the data virtualization process is finished. You'll now be able to inspect the result; to take a look at the objects, attributes and relations that make up your knowledge graph.

In addition, you can share it with specific workspaces or with all of them, and finally, you can use it in search strategies.