- Apache Spark is an open source cluster computing platform for data processing tasks
- It extends Apache Hadoop and introduces concepts such as stream processing (Spark Streaming) and Iterative Computation (Machine Learning tasks)
- Apache Spark was initially written in Scala and ships with a Scala and Python interactive shell – REPL(Read-Evaluate-Print-Loop). It includes the following APIs:
- Java
- Scala
- Python
Spark APIs: Java Vs Scala
Java
- Java less concise and more verbose and error prone – support for lambdas and stream only from Java 8
- More Established programming language, lots of experts in the market
- Full Java/Scala interoperability – implicit conversions between major Collections’ types
Scala
- Blend of functional and object-oriented aspects makes Scala highly scalable
- No distinction between an object and a function – every value is an object and every operation is a method call
- Scala’s type inference contributes to more readable programs
- Scala’s Traits tames multiple inheritance
- Scala displays conciseness, brevity and advanced static typing.
Scala API at work: Loading CSV Files
Java Api at work: Loading CSV Files
Final Remarks
- Scala’s strengths lay on Scalability, Conciseness and Advanced Static Typing together with full Java interoperability
- Scala can have a challenging learning curve and still has limited community presence (compared to Java and Python)
- Scala developers are still a niche Vs Rich Market for Java and Python professionals
- Python still a strong choice because of easy transition from OOP languages and the number of available statistical and Data Science libraries