apache / spark

# Apache Spark
Spark is a unified analytics engine for large-scale data processing. It provides
high-level APIs in Scala, Java, Python, and R (Deprecated), and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing,
and Structured Streaming for stream processing.
- Official version: 
- Development version: 
## Online Documentation
You can find the latest Spark documentation, including a programming
guide, on the project web page.
This README file only contains basic setup instructions.
## Build Pipeline Status
| Branch | Status |
|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| master | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-4.x | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-4.3 | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-4.2 | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-4.1 | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-4.0 | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| branch-3.5 | |
| | |
| | |
## Building Spark
Spark is built using Apache Maven.
To build Spark and its example programs, run:
 [code] 
(You do not need to do this if you downloaded a pre-built package.)
More detailed documentation is available from the project site, at
"Building Spark".
For general development tips, including info on developing Spark using an IDE, see "Useful Developer Tools".
## Interactive Scala Shell
The easiest way to start using Spark is through the Scala shell:
 [code] 
Try the following command, which should return 1,000,000,000:
 [code] 
## Interactive Python Shell
Alternatively, if you prefer Python, you can use the Python shell:
 [code] 
And run the following command, which should also return 1,000,000,000:
 [code] 
## Example Programs
Spark also comes with several sample programs in the `examples` directory.
To run one of them, use `./bin/run-example [params]`. For example:
 [code] 
will run the Pi example locally.
You can set the MASTER environment variable when running examples to submit
examples to a cluster. This can be spark:// URL,
"yarn" to run on YARN, and "local" to run
locally with one thread, or "local[N]" to run locally with N threads. You
can also use an abbreviated class name if the class is in the `examples`
package. For instance:
 [code] 
Many of the example programs print usage help if no params are given.
## Running Tests
Testing first requires building Spark. Once Spark is built, tests
can be run using:
 [code] 
Please see the guidance on how to
run tests