Amazon Redshift Integration For Apache Spark: ‘Fast And Seamless’

Amazon Redshift is now integrated with Apache Spark to help data engineers build and run Spark applications that can consume and write data from an Amazon Redshift cluster.

“Today if you’re working in EMR, you can use Spark to run analytics on data. But if you want to run a Spark query for data located in Redshift, you have to either move the data into S3 or find, download, and configure slow open source container to connector to Redshift. A better way would be to just run a Spark query on the data right in Redshift,” said Selipsky in his keynote. “So we wanted to make fast and seamless and I’m really excited to introduce Amazon Redshift integration for Apache Spark.”

Amazon Redshift integration for Apache Spark can now help developers seamlessly build and run Apache Spark applications on Amazon Redshift data.

If customers are using AWS analytics and machine learning services—such as Amazon EMR, AWS Glue, and Amazon SageMaker—they can now build Apache Spark applications that read from and write to their Amazon Redshift data warehouse without compromising on the performance of applications or transactional consistency of data.

“Now it’s incredibly easy to run Apache Spark applications on rRedshift data from AWS analytic services,” said the AWS CEO. “There’s now no more need to move any data, no need to build or manage any connectors.”

Selipsky said Amazon Redshift integration for Apache Spark minimizes the cumbersome and often manual process of setting up a Spark-Redshift open-source connector and reduces the time needed to prepare for analytics and ML tasks.