same as others big data technology,CheckPoint is a well-knowed solution to keep data a snapshot for speeduping failovers,ie. restores to most recent checkpoint state of data ,so u will not need to recomputate the rdd against the job.
in fact,the checkpoint op will cut down the relationships of all parent rdds.so the current rdd will be the last rdd of data line,and it will be derived by CheckpointRDD to achieve this goal.moreover,CheckpointRDDData is a other wrapper of CheckpointRDD.
1.how to
in spark,the checkpoint version is done by below steps(spark 1.4.1):
a. setup checkpoint dir by SparkContext.setupCheckpointDir(xx) b. snapshot a data state of timeline:rdd.checkpoint() c. do real checkpoint op at the last of a job(by default)
now lets detail more the steps respectively.
in the step 'b',the src is implemented by below codepath:
/** * Mark this RDD for checkpointing. It will be saved to a file inside the checkpoint * directory set with SparkContext.setCheckpointDir() and all references to its parent * RDDs will be removed. This function must be called before any job has been * executed on this RDD. It is strongly recommended that this RDD is persisted in * memory, otherwise saving it on a file will require recomputation.-cmp RDDCheckpointData#doCheckpoint() */ def checkpoint() { if (context.checkpointDir.isEmpty) { throw new SparkException("Checkpoint directory has not been set in the SparkContext") } else if (checkpointData.isEmpty) { checkpointData = Some(new RDDCheckpointData(this)) checkpointData.get.markForCheckpoint() } }
in the comment,u will curious about :why its necessary to persist the rdd,and to memory?
by diving into the src we know that the checkpoint op is really a job to run one more time on this rdd to save the result to file,so u will do one more computation if this rdd is not persisted.
on the other hand,why this rdd is recommanded to save in memory but disk? in fact,it's a little bit of differencs between the data saved in memory and file(maybe data format is),therefor,i think the author does not emphasize where to persist but the op of 'persist'.
2.FAQ
a.how to use checkpoint to restore data
from the StreamContext,we know that a func named 'getOrCreate(...)' is there for using the specified checkpoint dir defined before .so the snapshot data will readin rdd if any.
b.why not to save computated results when the rdd is run in first time
hm...no doubt,the real meaning of checkpont op is a second same job run on thie rdd.so why not to save ths results to file simetaneously at the first time?
first,there is only one anomyous function only defined in any runJob(..),thereby no more param can be accpted besides the user function .
second,the user function divided by the checkpoint save-op is more clearly to debug ,mantain etc.
相关推荐
python库。 资源全名:tensorflow-checkpoint-reader-0.1.0.tar.gz
fence-virtd-checkpoint-0.2.3-9.el6.i686.rpm是工具centos包 。
2018-09-06-checkpoint.ipynb
r50-deformable-detr-checkpoint.pth
store-sales-analysis-checkpoint.ipynb
数据库连接2022-12-1-checkpoint.ipynb
incubator-seatunnel-2.0.0-pre.zip: 源码可以编译打包,放入waterdrop dir的lib...flink-shaded-hadoop-2-uber-2.8.3-9.0.jar:flink checkpoint hadoop依赖,将flink-shaded-hadoop-2-uber.jar放入flink dir的lib下
First Order Model (512x512 pixels)模型文件 —— https://github.com/adeptflax/motion-models
实验指导书-1-checkpoint-1.ipynb
Exp1-产生式系统-todo-checkpoint(1).ipynb
CheckPoint_156-215.80 V18.75.pdf
2021年CheckPoint CCSA_156-215.80 V18.75.vce,题库
该模型使用rtx 3090 训练了5 天。2 个 GPU 似乎并没有太大地加快进程,它只有大约 50% - 60% 的 GPU 使用率。使用 mp4 文件作为训练数据,png 格式太大。该模型处于第 95 个纪元。训练脚本在完成之前由于某种未知...
Python库是一组预先编写的代码模块,旨在帮助开发者实现特定的编程任务,无需从零开始编写代码。这些库可以包括各种功能,如数学运算、文件操作、数据分析和网络编程等。Python社区提供了大量的第三方库,如NumPy、...
Python库是一组预先编写的代码模块,旨在帮助开发者实现特定的编程任务,无需从零开始编写代码。这些库可以包括各种功能,如数学运算、文件操作、数据分析和网络编程等。Python社区提供了大量的第三方库,如NumPy、...
Python库是一组预先编写的代码模块,旨在帮助开发者实现特定的编程任务,无需从零开始编写代码。这些库可以包括各种功能,如数学运算、文件操作、数据分析和网络编程等。Python社区提供了大量的第三方库,如NumPy、...
Python库是一组预先编写的代码模块,旨在帮助开发者实现特定的编程任务,无需从零开始编写代码。这些库可以包括各种功能,如数学运算、文件操作、数据分析和网络编程等。Python社区提供了大量的第三方库,如NumPy、...
CheckPoint 156-315.80q469.vce
CheckPoint-CCSE 176 question ETE格式.ete
python库。 资源全名:speaker-verification-0.1.2.tar.gz