# Data Explorer

The Datopian Data Explorer is a React single page application and framework for creating and displaying rich data explorers (think Tableau-lite). Use stand-alone or with CKAN. For CKAN it is a drop-in replacement for ReclineJS in CKAN Classic.

Data Explorer

Data Explorer for the City of Montreal

# Features / Highlights

“Data Explorer” is an embedable React/Redux application that allows users to:

  • Explore tabular, map, PDF, and other types of data
  • Create map views of tabular data using the Map Builder
  • Create charts and graphs of tabular data using Chart Builder
  • Easily build SQL queries for Data Store API using graphical interface of Datastore Query Builder

# Components

The Data Explorer application acts as a coordinating layer and state management solution – via Redux – for several libraries, also maintained by Datopian.

# Datapackage Views

Datapackage Views

Datapackage View is the rendering engine for the main window of the Data Explorer.

The above image displays a table shown at the Table tab, but note that Datapackage-views renders all data visualizations: Tables, Charts, Maps, and others.

# Datastore Query Builder

Datastore Query Builder

The Datastore Query Builder interfaces with the Datastore API to allow users to search data resources using an SQL like interface. See the docs for this module here - Datastore Query Builder docs.

# Map Builder

Map Builder

Map Builder allows users to build maps based on geo-data contained in tabular resources.

Supported geo formats:

  • lon / lat (separate columns)

# Chart Builder

Chart Builder

Chart Builder allows users to create charts and graphs from tabular data.

# Quick-start (Sandbox)

  • Clone the data explorer
$ git clone [email protected]:datopian/data-explorer.git
  • Use yarn to install the project dependencies
$ cd data-explorer
$ yarn
  • To see the Data Explorer running in a sandbox environment run Cosmos
$ yarn cosmos

# Configuration

data-datapackage attribute may influence how the element will be displayed. It can be created from a datapackage descriptor.

# Fixtures

Until we have better documentation on Data Explorer settings, use the Cosmos fixtures as an example of how to instantiate / configure the Data Explorer.

# Serialized state

store->serializedState is a representation of the application state without fetched data
A data-explorer can be “hydrated” using the serialized state, it will refetch the data, and will render in the same state it was exported in

Share links can be added in datapakage.resources[0].api attribute.

There is common limit of up 2000 characters on URL strings. Our share links contain the entire application store tree, which is often larger than 2000 characters, in which the application state cannot be shared via URL. Thems the breaks.

# Translations

# Add a Translation To Data Explorer

To add a translation to a new language to the data explorer you need to:

  1. clone the repository you need to update
git clone [email protected]:datopian/data-explorer.git
  1. go to src/i18n/locales/ folder
  2. add a new sub-folder with locale name and the new language json file (e.g. src/i18n/locales/ru/translation.json)
  3. add the new file to resources settings in i18n.js:
    src/i18n/i18n.js:
import en from './locales/en/translation.json'
import da from './locales/da/translation.json'
import ru from './locales/ru/translation.json'
    ...
      ru: {
        translation: {
          ...require('./locales/ru/translation.json'),
          ...
        }
      },
    ...
  1. create a merge request with the changes

# Add a translation To a Component

Some strings may come from a component, to add translation for them will require some extra steps, e.g. datapackage-views-js:

  1. clone the repository
https://github.com/datopian/datapackage-views-js.git
  1. go to src/i18n/locales/ folder
  2. add a new sub-folder with locale name and the new language json file (e.g. src/i18n/locales/ru/translation.json)
  3. add the new file to resources settings in i18n.js:
    src/i18n/i18n.js:
...
import ru from './locales/ru/translation.json'
    ...
    resources: {
      ...
      ru: {translation: ru},
    },
    ...
  1. create a pull request for datapackage-views-js
  2. get the new datapackage-views-js version after merging (e.g. 1.3.0)
  3. clone data-explorer
  4. upgrade the data-explorer’s datapackage-views-js dependency with the new version:
    a. update package.json
    b. run yarn install
  5. add the component’s translations path to Data Explorer:
import en from './locales/en/translation.json'
import da from './locales/da/translation.json'
import ru from './locales/ru/translation.json'
    ...
      ru: {
        translation: {
          ...require('./locales/ru/translation.json'),
          ...require('datapackage-views-js/src/i18n/locales/ru/translation.json'),
        }
      },
    ...
  1. create a merge request for data-explorer

# Testing a Newly Added Language

To see your language changes in Data Explorer you can run yarn start and change the language cookie of the page (defaultLocale):

i18n Cookie

# Language detection

Language detection rules are determined by detection option in src/i18n/i18n.js file. Please edit with care, as other projects may already depend on them.

# Embedding in CKAN NG Theme

# Copy bundle files to theme’s public directory

$ cp data-explorer/build/static/js/*.js frontend-v2/themes/your_theme/public/js
$ cp data-explorer/build/static/js/*.map frontend-v2/themes/your_theme/public/js
$ cp data-explorer/build/static/css/* frontend-v2/themes/your_theme/public/css

# Note on app bundles

The bundled resources have a hash in the filename, for example 2.a3e71132.chunk.js

During development it may be preferable to remove the hash from the file name to avoid having to update the script tag during iteration, for example

$ mv 2.a3e71132.chunk.js 2.chunk.js

A couple caveats:

  • The .map file names should remain the same so that they are loaded properly
  • Browser cache may need to be invalidated manually to ensure that the latest script is loaded

# Require Data Explorer resources in NG theme template

In /themes/your-theme/views/your-template-wth-explplorer.html

<!-- Everything before the content block goes here -->
{% block content %}

<!-- Data Explorer CSS -->
<link rel="stylesheet" type="text/css" href="/static/css/main.chunk.css">
<link rel="stylesheet" type="text/css" href="/static/css/2.chunk.css">
<!-- End Data Explorer CSS -->

# Configure datapackage

<!-- where datapackage is -->
<srcipt>
  const datapackage = {
    resources: [{resource}], // single resource for this view
    views: [...], // can be 3 views aka widgets
    controls: { 
      showChartBuilder: true,
      showMapBuilder: true 
    }
  }
</srcipt>

# Add data-explorer tags to the page markup

Each Data Explorer instance needs a corresponding <div> in the DOM. For example:

{% for resource in dataset.resources %}
  <div class="data-explorer" id="data-explorer-{{ loop.index - 1 }}" data-datapackage='{{ dataset.dataExplorers[loop.index - 1] | safe}}'></div>
{% endfor %}

Note that each container div needs the following attributes:

  • class="data-explorer" (All explorer divs should have this class)
  • id="data-explorer-0" (1, 2, etc…)
  • data-datapackage={JSON CONFIG}` (A valid JSON configuration)

# Add data explorer scripts to your template

<script type="text/javascript" src="/static/js/runtime~main.js"></script>
<script type="text/javascript" src="/static/js/2.chunk.js"></script>
<script type="text/javascript" src="/static/js/main.chunk.js"></script>

NOTE that the scripts should be loaded after the container divs are in the DOM, typically by placing the <script> tags at the bottom of the footer

See a real-world example here

# New builds

In order to build files for production, run npm run build or yarn build.

You need to have node version >= 12 in order to build files. Otherwise a ‘heap out of memory error’ gets thrown.

# Appendix: Design

See Data Explorer Design page »