Testing a Spark Application

09 May 2017

Data engineering / analysis applications are just like any other software. You need to test them. But how?

I wrote something over at the OpenCredo blog about Testing Spark.

It covers a few different technologies, mostly focused around data engineering (Cassandra, Spark, Scala). The thrust of it is that you can test any kind of application. Particularly data processing applications, which traditionally aren’t built by software engineers, and don’t tend to be tested.