Blog entries

Tensei-Data 1.9.1 Announcement

Submitted by Jens Grassel on Tue, 11/22/2016 - 18:38

We just released version 1.9.1 of Tensei-Data.

This is a bugfix release one that is highly recommended if you use triggers. Version 1.9.0 introduced a regression that could under some circumstances lead to the behaviour that triggers are executed not once but multiple times.

To update the demo version you can use the following sequence of commands. Please export your cookbooks before doing this if you don't want to loose them.

% vagrant box update
% vagrant destroy
% vagrant up

Easier installation of Tensei-Data demo version

Submitted by Jens Grassel on Thu, 11/17/2016 - 13:18

The installation of the Tensei-Data demo version just became a lot easier.

We decided to make the vagrant box public. Running the demo now is as simple as creating a new folder and issuing the following two commands:

vagrant init wegtam/tensei-demo
vagrant up --provider virtualbox

After the box has been started, just point your browser to http://localhost:9000.

Tensei-Data 1.9.0 Announcement

Submitted by Jens Grassel on Fri, 11/11/2016 - 10:09

Version 1.9.0 contains a lot of bug fixes and an advanced permission system for the frontend which is a long desired feature for users that have lots of configurations and users.

Summary of Changes

Every component of the system now has a file conf/application.ini that can be used for runtime configuration like memory usage. You don't have to modify the agent startup script parameters anymore!

The permission system for the frontend allowes fine grained access and usage permissions which should suffice for any usecases in environments that need permissions. All resources now have an owner, a possible owner group and permissions that define the rights for owner, group and any other (world). It resembles the traditional Unix permissions.

Added

  • application.ini for runtime configuration options
  • defaults for logback configuration options
  • support for Firebird database
  • simple transformer for converting strings into long numbers
  • advanced permission system based on users, groups and base permissions

Changed

  • update Akka to 2.4.12
  • update several database drivers
  • more fine grained logging configuration

Fixed

  • UniqueValueBuffer does not unsubscribe from event channel after shutdown
  • AutoIncrementValueBuffer does not unsubscribe from event channel after shutdown
  • Tests using sqlite do not run under FreeBSD.
  • Broken floating point number parsing on some databases (data corruption!).
  • Wrong format descriptions generated for some floating point values and decimals.
  • agent run logs loaded in endless loop
  • loglines saved multiple times (duplicated) into buffer table
  • crash when importing cookbook with same source and target dfasdl
  • cookbooks not completely deleted from database
  • prevent deletion of cookbooks used by transformation configurations
  • no work history shown on dashboard if no agent connected
  • work history not loaded correctly for multiple transformations
  • triggering transformations by other transformations starts wrong transformation

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

New design for the Wegtam GmbH Homepage

Submitted by Andre Schuetz on Thu, 06/23/2016 - 12:09

This new website integrates all products of the Wegtam GmbH in one spot. We inform you about our products, releases, participations and further engagements.

Furthermore, the former blog of wegtam.net will be integrated into this website. The former free online search of https://www.wegtam.com will be relaunched at https://www.wegtam.net.

This is the basis for the coming developments and improvements for all of our products and services.

Tags

Tensei-Data 1.8.0 Announcement

Submitted by Jens Grassel on Wed, 06/22/2016 - 15:38

With version 1.8.0 of Tensei-Data we provide data normalisation and deduplication functionality! The foundation for that is the support of unique constraints and more foreign key value types.

Summary of changes

There are lots of small changes und improvements all over the place. The main focus in this release is support for unique constraints and more advanced foreign key support. This provides the basis for data normalisation and deduplication.

Added

  • support for normalisation
  • support more foreign key field types
  • support optional values for foreign keys e.g. empty (NULL) values
  • support for unique constraints
  • support checking for existing unique values in target database
  • extract schema from Derby
  • extract schema from MariaDB
  • extract primary key, foreign key and auto increment information from HyperSQL
  • extract primary key, foreign key and auto increment information from Microsoft SQLServer
  • date field support for Oracle
  • line break support for csv files
  • send log files to frontend upon request
  • transformer for converting datetime values
  • show previous version of DFASDL including diff
  • better error handling of frontend errors
  • fetch logs from agent upon request

Changed

  • switched log message formatting to a more performant solution
  • separate logfiles for each run
  • added current date and time support to overwrite transformer
  • internal data hashing algorithm (53 times faster and 4/5 less memory usage!)
  • several ui fixes and cleanups

Removed

  • logging to database

Fixed

  • agent becomes unauthorised from time to time
  • use element reference instead of id
  • cluster client reconnect problems
  • crash when extracting H2 database schema
  • ignore fulltext search columns for PostgreSQL databases
  • stop database writer if an error occurs
  • recognise formatnum when parsing csv
  • cronjob issues
  • ORA-00911: invalid character
  • csv file with empty lines at the end produce errors
  • error resolving foreign key relations
  • oracle does not support getMetaData and getParameterType
  • if then else numeric transformer error
  • editor not working correctly in Firefox
  • exception of database logging adapter

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

Tensei-Data 1.7.0 Announcement

Submitted by Jens Grassel on Thu, 03/03/2016 - 15:35

"Referential integrity", yes we did it! With Tensei-Data 1.7.0 we have included support for referential integrity. This means that you don't need to worry any longer if you're writing into a database that uses auto-increment columns for values that are used within foreign keys. You don't even need to do a single thing. If the DFASDL description of the database is correct then Tensei-Data will automatically take the actions necessary to keep integrity.

Summary of changes

The main feature in this release is the support of referential integrity of foreign keys in combination with auto-increment database columns. In addition to that we fixed several bugs, did some cleanup and added more documentation.

Added

  • Keep referential integrity when using auto-increment columns
  • Display version information in frontend
  • Generate DFASDL with correct auto-increment and foreign key elements
    • Currently only for H2, MySQL and PostgreSQL! More to come!
  • DateTypeConverter transformer
  • Recognition of several date and time formats for DFASDL auto-creation
  • Support for creating vagrant boxes for VMWare (on request)
  • Loading animation on time intense operations in the mapping editor
  • Lots of small improvements to documentation

Changed

  • Name of auto generated DFASDL defaults to CHANGEME
  • Attribute xml-element-name is now allowed on all DFASDL elements
  • Increased performance of mapping editor

Fixed

  • Suggester creating too many recipes
  • ID-Transformer crashing
  • Mapping-Key in the editor sometimes not saved correctly
  • Oracle-JDBC ORA-01882: timezone region not found (see documentation!)
  • Import and export of huge cookbooks was broken
  • Cookbook overview page loading very slow
  • Associativity between OneToOne and AllToAll mappings broken
  • Use formatnum instead of num for numbers with leading zeros
  • Parser crashes on xml including empty elements
  • Problems with Oracle data fields

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

Tensei-Data 1.6.0 Announcement

Submitted by Jens Grassel on Thu, 01/21/2016 - 13:34

Version 1.6.0 of the Tensei-Data migration software was just released. This time you get a lot of bugfixes, more robustness, the groundwork for very interesting future features and a breaking change!

WARNING: Breaking changes! This release contains changes that will render saved cookbooks invalid! Please see the migration section below.

Summary of changes

This is primary a release containing a lot of bugfixes and some major refactoring and to our regret a breaking change in the cookbook format.

Added

  • Support for SQLite databases.
  • Documentation of stop sign handling for windows machines added to DFASDL specification.
  • Output of the current byte offset if a parser exception occurs.
  • Documentation for manual installation outside of a virtual machine.

Changed

  • Automatic cancellation of the current job if a writer dies.
  • Refactoring of the whole processor architecture which allows more robust error handling and will lead to parallel execution in a future release!
  • Handling of element ids was problematic therefore it was replaced with "element references" which leads to a breaking change in the cookbook format.

Cookbook migration

The serialised cookbook is in link:http://www.json.org[JSON] format. There are 2 sections that need to be adapted to be able to load an old cookbook.

Before an element was referenced simply by the id. From now own an element reference is created that holds the id of the DFASDL and the id of the element within the DFASDL.

Atomic Transformations

If your cookbook contains atomic transformations then it will currently look like this:

Old json format

{
  "elementId": "ID",
  ...
}

The "elementId" part has to be renamed and changed to a reference like this:

New json format

{
  "element": {
    "dfasdlId": "DFASDL-ID", //<1>
    "elementId": "ELEMENT-ID" //<2>
  },
  ...
}

<1> The id of the DFASDL that contains the element. <2> The id of the element within the DFASDL.

Mappings

The mappings hold now complete element references instead of simple ids. Before they looked like this:

Old json format

"mappings": [
  {
    "sourceIds": [ "S-ID1", "S-ID2" ],
    "targetIds": [ "T-ID1", "T-ID2" ],
    ...
  },
  ...
]

Please rename the attribute sourceIds to sources and the attribute targetIds to targets. Their content has to be transformed to look like this:

New json format

"mappings": [
  {
    "sources": [
      {
        "dfasdlId": "SOURCE-DFASDL-ID", // The id of the source DFASDL that contains the element.
    "elementId": "S-ID1"
      },
      {
        "dfasdlId": "SOURCE-DFASDL-ID",
    "elementId": "S-ID2"
      }
    ],
    "targets": [
      {
        "dfasdlId": "TARGET-DFASDL-ID", // The id of the target DFASDL that contains the element.
    "elementId": "T-ID1"
      },
      {
        "dfasdlId": "TARGET-DFASDL-ID",
    "elementId": "T-ID2"
      }
    ],
    ...
  },
  ...
]

Fixed

  1. Consideration of a CSV header line for an auto generated DFASDL.
  2. Possible NullPointerException on invalid connection information URI.
  3. No automatic override of resources upon cookbook import.
  4. Wrong number of pages in log message pager.
  5. Parser exception when parsing CSV files with last column data missing.
  6. Possibly wrong auto generated DFASDL from CSV files.
  7. Disable of profiling in production code.
  8. Recognition of source data encoding for auto generated DFASDL documents.
  9. Too strong license limitations that prevented some users from creating cookbooks and transformation configurations.
  10. Auto generated DFASDL elements from CSV files could contain invalid characters.
  11. Display errors upon DFASDL elements with the same id in the cookbook editor.

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

Tensei-Data 1.5.0 Announcement

Submitted by Jens Grassel on Tue, 12/01/2015 - 13:33

Version 1.5.0 of the Tensei-Data migration software was just released. This release brings you some fixes and improvements in the frontend, special support for JSON files and a virtual machine that scales!

Summary of changes

The main feature in this release might be the automatic scaling of the virtual machine. Until now the demo vm could not benefit from more resources. But not any longer! From now on the system components will use more memory if it is provided.

This enables us to reduce the requirements for the demo vm from 5 GB down to 2-3 GB. And further more it enables you to give more memory to the machine if you run memory intense transformations.

TIP: Usually 4-5 GB of RAM should suffice for almost anything. But you're free to try to max things out.

Added

  • Special support for JSON files including automatic DFASDL generation.
  • Automatic scaling of the virtual machine.

Changed

  • Cleaned up the visibility section in the editor.
  • During automatic DFASDL generation a working indicator is displayed now.

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

Tensei-Data 1.4.2 Announcement

Submitted by Jens Grassel on Tue, 10/13/2015 - 13:31

We just released version 1.4.2 of the Tensei-Data migration software. Like the last release this is a bugfix release and all users are advised to upgrade.

Summary of changes

In addition to the bugfixes some minor improvements are released with this version.

Added

  • The tooltips are now showing the transformations and the mappings.
  • Highlighting of the currently selected node in the editor when creating mappings.
  • If a MapAllToAll mapping is created the source node is not de-selected after the first target node is clicked. You can now create all-to-all mappings much more swiftly.

Changed

  • The mapping lines are now drawn "behind" the rest of the editor pane increasing readability.

Bugfixes

  • Some ids were missing in the cookbook editor tooltips.
  • Long names were not wrapped and cluttered up the editor window.
  • Mappings in the left side of the editor pane could not be deleted.

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.

Tensei-Data 1.4.1 Announcement

Submitted by Jens Grassel on Thu, 10/01/2015 - 13:30

Version 1.4.1 of the Tensei-Data migration software has just been released. This is a bugfix release and all users using atomic transformers should upgrade.

In addition to the bugfix you get some performance and usability improvements.

Summary of changes

Bugfixes

  • The configuration for atomic transformers was not saved correctly leading to unexpected behaviour and errors.
  • Avoid drawing of editor panes multiple times.

Changes

  • The element ids are shortened to make them easier to read.
  • More compact design in the editor.
  • Show transformers on the left side of the editor.

Get your free trial version

You can try the Tensei-Data system for free. Just request a free trial version and receive the download link and a 14 days free license.