Jekyll2024-11-16T21:27:04+00:00https://datacleaner.github.io/feed.xmlThe premier open source Data Quality solution | DataCleanerOpen source software for data quality, data profiling, data warehousing, data wrangling, master data management, business intelligence and governance. DataCleaner 5.8.1 released2022-02-09T21:28:00+00:002022-02-09T21:28:00+00:00https://datacleaner.github.io/release/2022/02/09/datacleaner-5.8.1-releasedHello DataCleaner community! It has been over a year since our last release, but we finally have exciting news to share. DataCleaner version 5.8.1 has just been made available!

The main news to share is that DataCleaner now runs on all the latest version of Java, which required us to do a whole lot of modernizing of the platform that the application is built on. A total of more than 20 libraries have been updated and fitted to work with the latest Java runtimes.

In addition to this critical work, we’ve also rewired the RegexSwap catalog of regular expressions that you can use. The old RegexSwap web services had been defunct for a while, but with a new API endpoint that is controlled by the community, we’re online with reusable regular expressions again!

You can get DataCleaner version 5.8.1 on the Downloads page!

]]>DataCleaner 5.7.1 released2020-05-27T21:12:00+00:002020-05-27T21:12:00+00:00https://datacleaner.github.io/release/2020/05/27/datacleaner-5.7.1-releasedWe’re happy to announce the release of DataCleaner version 5.7.1. This release is a minor update that addresses a couple of issues and improvements primarily found in libraries and other systems that we depend on.

#1849 - Fixed compatibility with recent versions of PostgreSQL.
#1825 - Improved error messages when JDBC connection is failing
Upgraded several dependencies, incl. Apache MetaModel, Apache Spark, Apache Hadoop, Jackson and Scala.

You can get the new version on the Downloads page.

]]>DataCleaner 5.7.0 released2019-04-01T19:22:00+00:002019-04-01T19:22:00+00:00https://datacleaner.github.io/release/2019/04/01/datacleaner-5.7.0-releasedNo, this is not an April’s fool’s post. We’re releasing DataCleaner 5.7.0 today and it’s pretty cool stuff coming out!

Grab it here – Downloads – or keep reading for more information.

Machine Learning

DataCleaner 5.7.0 includes half a dusin new components all resolving around the topic of Machine Learning. As the data quality engineer’s toolkit, we found it naturally to also expand DataCleaner’s applicability into the realm of ML. With DataCleaner 5.7.0 you can thus train and apply ML models on every data source that DataCleaner supports.

Grab the software or check out the Documentation for details about the new components.

Amazon AWS DynamoDB

We’ve added support for Amazon AWS DynamoDB. So for those cloud database users out there, you’ve no longer got an excuse not to use DataCleaner.

More stuff

We’ve added a new String contains filter component which lets you easily filter out records that contains particular text snippets.

A bug was fixed in the Pattern Finder analyzer, which caused it’s initial configuration for parsing/identifying numbers to be locale-dependent.

]]>Human Inference acquired by EDM2019-02-11T15:41:00+00:002019-02-11T15:41:00+00:00https://datacleaner.github.io/www/update/2019/02/11/human-inference-acquired-by-edmToday we saw an announcement from Neopost that they are divesting from the Data Quality space and have sold Human Inference to the Dutch company EDM Media Group. With this sale, the DataCleaner commercial editions will get a new home with EDM instead of Neopost/Quadient.

The open source community editions of DataCleaner (provided via this site) are unaffected. DataCleaner is still an open source project on GitHub. We congratulate the teams at Human Inference for a successful acquisition by a new owner that will no doubt be great tenants to the commercial brand of DataCleaner.

As for the open source community, we continue on in delivering the greatest freely avaliable Data Quality solution available. You may be wondering about the roadmap for DataCleaner community edition, and it just happened to be that this weekend we opened this Twitter poll to gauge the interest for various features in the community.

Please help provide your feedback on this topic.

We’ve also just opened a new Gitter chat channel which will allow community members to have a higher degree of interaction. Several community members have a past work-relationship either in Human Inference, Neopost or Quadient. With the Gitter chat we hope to provide a way for people to keep in touch and to continue collaborating on the future direction of DataCleaner.

]]>DataCleaner 5.6.0 released2019-01-28T05:47:00+00:002019-01-28T05:47:00+00:00https://datacleaner.github.io/release/2019/01/28/datacleaner-5.6.0-releasedWe’ve cut another release of DataCleaner, modernizing your favourite open source Data Quality tool to be able to keep up with the latest database versions, the latest Java distributions and so on.

This release doesn’t change much in terms of features and functions in the application itself, but it DOES make sure that everything works also on the latest technology. The release upgrades several key dependencies:

  • Apache POI (used for reading Excel spreadsheets) is now at version 4.
  • Apache Hive is now at version 2.
  • Apache Hadoop is now at version 3.
  • Apache HBase is now at version 2.

In addition, several internal libraries used have been upgraded to ensure that DataCleaner runs smoothly on all the available Java runtimes out there, and at the best performance possible, too.

You can find DataCleaner 5.6.0 now at the Downloads page.

]]>DataCleaner 5.5.0 released2018-10-18T18:19:00+00:002018-10-18T18:19:00+00:00https://datacleaner.github.io/release/2018/10/18/datacleaner-5.5.0-releasedProfiling and transforming streaming data never was easier than with the new DataCleaner 5.5.0, now proudly consuming live data streams from Apache Kafka.

  • #1794 - Added connectivity to Apache Kafka.

Get the new DataCleaner download at the Downloads page.

]]>DataCleaner 5.4.2 released2018-05-29T05:23:00+00:002018-05-29T05:23:00+00:00https://datacleaner.github.io/release/2018/05/29/datacleaner-5.4.2-releasedOnce again there’s a small software update from the DataCleaner community to the DataCleaner community.

This time around DataCleaner 5.4.2 is available and it has a couple of good bugfixes and feature improvements up it’s sleeve:

  • #1790 - Improved the component search under the library tree.
  • #1787 - Improved resizing and high-dpi support for file selection/browsing widget.
  • #1789 - Upgraded JUNG graph API to 2.1.1.
  • #1788 - Fixed compatibility with JDK 10.

Get the new DataCleaner download at the Downloads page.

]]>DataCleaner 5.4.0 released2018-01-23T01:20:00+00:002018-01-23T01:20:00+00:00https://datacleaner.github.io/release/2018/01/23/datacleaner-5.4.0-releasedWe’re happy to announce the availability of DataCleaner Community Edition version 5.4.0! You can grab it on the Downloads page now. Let’s jump right into what’s new:

New analyzer: “Mark rows as…”

We’ve added the analyzer “Mark rows as …” to DataCleaner (issue #942). This is a component that allows you to consume data, present a preview of it, and count records passed to it. A very basic component that lends itself to advanced use cases such as complex filtering, error handling and rule/metric monitoring.

More information in the Component Library.

Software update notifications

Going forward, DataCleaner will notify you (in the lower right corner of the status bar) of new versions available (issue #1009).

Minor bugfixes

We’ve addressed a series of minor bugs that applied to the previous versions:

]]>DataCleaner 5.3.0 released2017-11-23T04:20:00+00:002017-11-23T04:20:00+00:00https://datacleaner.github.io/release/2017/11/23/datacleaner-5.3.0-releasedOver the last couple of months, DataCleaner community members have been working to shape together a new release of DataCleaner that will shape the application and it’s source code in a way that is more sustainable from a community and contributor perspective. Today, this release is available. Go to the Downloads page to get DataCleaner 5.3.0!

Or read on to learn about what’s new …

Sanitized the source for contributor friendliness

As the community edition and commercial edition of DataCleaner are now effectively separated, we decided to go ahead and cut some of the ties and remove complexity that is needed for the enterprise and highly customizable scenarios that DataCleaner have had to cater to, but no longer applies to. Specifically, a number of features aren’t available in DataCleaner community edition, which allowed us to trim the codebase considerably:

  • The on-demand services from DataCloud are no longer included.
  • The integration with DataCleaner monitor is no longer included.
  • The ExtensionSwap and RegexSwap content providers are no longer included.
  • Deprecated code and backwards compatibility mode for versions prior to DataCleaner 5.0 has been dropped.

It may not seem exciting to just remove things. But for potential contributors, this is a big deal. We’ve cut away more than 18,000 lines of code from the project! This makes it leaner and easier to work with. And we’re very happy to have a less involved development pipeline for the community.

New feature: Fill pattern analyzer

Do you wonder which fields are filled, and depending on what? Is the “state” field of an address always filled, or does it depend on the country? The Fill pattern analyzer lets you discover these relationships.

In the screenshot above you see the distribution of how records are filled in a simple data set. Below you see a more complex scenario, where the filling of fields has a high variety of shapes.

This feature has been available as a third party extension for a while. Now it is fully integrated into the community edition distribution.

New feature: Groovy scripting transformer

Another feature that was added in this release, is the Groovy transformer. The new transformer allows you to use the Groovy programming language in your DataCleaner jobs. The transformation is comparable to the well-known JavaScript transformer, but the Groovy language is arguably both faster and more powerful, although JavaScript is perhaps more widely known. Now DataCleaner users have the benefit of choice as they select the scripting language that best fit their problem.

Basic support for high DPI screens

It has long been a painful experience to use DataCleaner on high DPI devices. With DataCleaner 5.3.0 we’ve added basic support for high DPI screens. The support here is “basic” in the sense that there are still areas of improvements, but a big step forward has been taken, and the framework has been built for future enhancements too.

]]>DataCleaner 5.2.2 released2017-08-25T04:20:00+00:002017-08-25T04:20:00+00:00https://datacleaner.github.io/release/2017/08/25/datacleaner-5.2.2-releasedToday we’ve released version 5.2.2 of DataCleaner community edition. This release is a rather small one, but was cut due to the blocking nature of one of the handled issues. Here’s a summary:

  • Fixed a bug which caused DataCleaner not to be able to start on Linux or Mac environments when launched via the datacleaner.sh script.
  • Updated our dependency on the data access library Apache MetaModel to their latest version (5.0-RC3) to reap the benefits of several stabilization and bug fixes.

Go to the Downloads page to get DataCleaner 5.2.2!

]]>