This document gives a short overview of how onfire-bg.com runs on clusters, to make it easier to lớn understandthe components involved. Read through the application submission guideto learn about launching applications on a cluster.
Bạn đang xem: Chế độ manager 2.0Components
onfire-bg.com applications run as independent sets of processes on a cluster, coordinated by the onfire-bg.comContextobject in your main program (called the driver program).
Specifically, to lớn run on a cluster, the onfire-bg.comContext can connect to lớn several types of cluster managers(either onfire-bg.com’s own standalone cluster manager, Mesos or YARN), which allocate resources acrossapplications. Once connected, onfire-bg.com acquires executors on nodes in the cluster, which areprocesses that run computations & store data for your application.Next, it sends your application code (defined by JAR or Pythuôn files passed khổng lồ onfire-bg.comContext) tothe executors. Finally, onfire-bg.comContext sends tasks lớn the executors lớn run.
There are several useful things lớn note about this architecture:Each application gets its own executor processes, which stay up for the duration of the wholeapplication và run tasks in multiple threads. This has the benefit of isolating applicationsfrom each other, on both the scheduling side (each driver schedules its own tasks) và executorside (tasks from different applications run in different JVMs). However, it also means thatdata cannot be shared across different onfire-bg.com applications (instances of onfire-bg.comContext) withoutwriting it lớn an external storage system. onfire-bg.com is agnostic khổng lồ the underlying cluster manager. As long as it can acquire executorprocesses, and these communicate with each other, it is relatively easy to lớn run it even on acluster manager that also supports other applications (e.g. Mesos/YARN). Because the driver schedules tasks on the cluster, it should be run cthất bại khổng lồ the workernodes, preferably on the same local area network. If you’d like lớn sover requests to thecluster remotely, it’s better to lớn open an RPC lớn the driver and have sầu it submit operationsfrom nearby than to run a driver far away from the worker nodes.Cluster Manager Types
The system currently supports several cluster managers:
A third-party project (not supported by the onfire-bg.com project) exists lớn add tư vấn forNomad as a cluster manager.Submitting Applications
Applications can be submitted to a cluster of any type using the onfire-bg.com-submit script.The application submission guide describes how khổng lồ do this.Monitoring
Each driver program has a website UI, typically on port 4040, that displays information about runningtasks, executors, & storage usage. Simply go to http://:4040 in a website browser toaccess this UI. The monitoring guide also describes other monitoring options.
onfire-bg.com gives control over resource allocation both across applications (at the level of the clustermanager) & within applications (if multiple computations are happening on the same onfire-bg.comContext).The job scheduling overview describes this in more detail.Glossary
The following table summarizes terms you’ll see used khổng lồ refer khổng lồ cluster concepts:
|Application||User program built on onfire-bg.com. Consists of a driver program and executors on the cluster.|
|Application jar||A jar containing the user"s onfire-bg.com application. In some cases users will want lớn create an "uber jar" containing their application along with its dependencies. The user"s jar should never include Hadoop or onfire-bg.com libraries, however, these will be added at runtime.|
|Driver program||The process running the main() function of the application và creating the onfire-bg.comContext|
|Cluster manager||An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)|
|Deploy mode||Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.|
|Worker node||Any node that can run application code in the cluster|
|Executor||A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.|
|Task||A unit of work that will be sent to one executor|
|Job||A parallel computation consisting of multiple tasks that gets spawned in response khổng lồ a onfire-bg.com action (e.g. save sầu, collect); you"ll see this term used in the driver"s logs.|
|Stage||Each job gets divided inkhổng lồ smaller sets of tasks called stages that depover on each other (similar to lớn the bản đồ and reduce stages in MapReduce); you"ll see this term used in the driver"s logs.|