CU13 brought some exciting new capabilities and important changes for SQL Server Big Data Clusters.
In brief, these were:
- Hadoop Distributed File System (HDFS) distributed copy capabilities through azdata
- Enable Advanced Encryption Standard (AES) Optional parameter on the automatically generated AD accounts
- Password rotation for Big Data Cluster’s auto-generated Active Directory service accounts during BDC deployment
- Apache Spark 3.1.2
- SQL Server Big Data Clusters runtime for Apache Spark release 2021.1
You can find out more about each of these capabilities and changes here: What’s new with SQL Server Big Data Clusters—CU13 Release – Microsoft SQL Server Blog
More About SQL Server Big Data Clusters
Released in 2019 as part of SQL Server 2019, SQL Server Big Data Clusters was the expansion SQL Server needed to allow it to embrace unstructured big data, relational data and scale-out compute.
Looking closely, you find that it is made up of a SQL Server database engine, Spark and Hadoop Distributed File System running on Kubernetes. As these all work together in a unified way, it enables you to read, write and process big data from Spark or T-SQL.