Chế độ manager 2.0

This document gives a short overview of how runs on clusters, to make it easier to lớn understandthe components involved. Read through the application submission guideto learn about launching applications on a cluster.

Bạn đang xem: Chế độ manager 2.0

Components applications run as independent sets of processes on a cluster, coordinated by the onfire-bg.comContextobject in your main program (called the driver program).

Specifically, to lớn run on a cluster, the onfire-bg.comContext can connect to lớn several types of cluster managers(either’s own standalone cluster manager, Mesos or YARN), which allocate resources acrossapplications. Once connected, acquires executors on nodes in the cluster, which areprocesses that run computations & store data for your application.Next, it sends your application code (defined by JAR or Pythuôn files passed khổng lồ onfire-bg.comContext) tothe executors. Finally, onfire-bg.comContext sends tasks lớn the executors lớn run.


There are several useful things lớn note about this architecture:

Each application gets its own executor processes, which stay up for the duration of the wholeapplication và run tasks in multiple threads. This has the benefit of isolating applicationsfrom each other, on both the scheduling side (each driver schedules its own tasks) và executorside (tasks from different applications run in different JVMs). However, it also means thatdata cannot be shared across different applications (instances of onfire-bg.comContext) withoutwriting it lớn an external storage system. is agnostic khổng lồ the underlying cluster manager. As long as it can acquire executorprocesses, and these communicate with each other, it is relatively easy to lớn run it even on acluster manager that also supports other applications (e.g. Mesos/YARN). Because the driver schedules tasks on the cluster, it should be run cthất bại khổng lồ the workernodes, preferably on the same local area network. If you’d like lớn sover requests to thecluster remotely, it’s better to lớn open an RPC lớn the driver and have sầu it submit operationsfrom nearby than to run a driver far away from the worker nodes.Cluster Manager Types

The system currently supports several cluster managers:

A third-party project (not supported by the project) exists lớn add tư vấn forNomad as a cluster manager.

Submitting Applications

Applications can be submitted to a cluster of any type using the script.The application submission guide describes how khổng lồ do this.


Each driver program has a website UI, typically on port 4040, that displays information about runningtasks, executors, & storage usage. Simply go to http://:4040 in a website browser toaccess this UI. The monitoring guide also describes other monitoring options.

Xem thêm: " Bộ Sưu Tập Tiếng Anh Là Gì, Bộ Sưu Tập Ảnh Tiếng Anh Là Gì

Job Scheduling gives control over resource allocation both across applications (at the level of the clustermanager) & within applications (if multiple computations are happening on the same onfire-bg.comContext).The job scheduling overview describes this in more detail.


The following table summarizes terms you’ll see used khổng lồ refer khổng lồ cluster concepts:

Application User program built on Consists of a driver program and executors on the cluster.
Application jar A jar containing the user"s application. In some cases users will want lớn create an "uber jar" containing their application along with its dependencies. The user"s jar should never include Hadoop or libraries, however, these will be added at runtime.
Driver program The process running the main() function of the application và creating the onfire-bg.comContext
Cluster manager An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)
Deploy mode Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.
Worker node Any node that can run application code in the cluster
Executor A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.
Task A unit of work that will be sent to one executor
Job A parallel computation consisting of multiple tasks that gets spawned in response khổng lồ a action (e.g. save sầu, collect); you"ll see this term used in the driver"s logs.

Xem thêm: Công Ty Tnhh Phát Triển Đô Thị Và Xây Dựng 379, Các Dự Án Của 379 Ud Co

Stage Each job gets divided inkhổng lồ smaller sets of tasks called stages that depover on each other (similar to lớn the bản đồ and reduce stages in MapReduce); you"ll see this term used in the driver"s logs.

Chuyên mục: Công Nghệ 4.0