feat(load): reject managed TsFile directories#18271
Conversation
| private static void validateNotLoadingInternalTsFile(final File file) | ||
| throws FileNotFoundException { | ||
| final Path sourcePath = canonicalPath(file); | ||
| for (final String dataDir : IoTDBDescriptor.getInstance().getConfig().getLocalDataDirs()) { |
There was a problem hiding this comment.
[P2] Hoist managed-directory canonicalization out of the recursive traversal
findAllTsFile calls this method for every file and directory it encounters, and every call re-canonicalizes both sequence and unsequence under every local data directory. Directory loading therefore performs 1 + 2 * dataDirCount canonical-path resolutions per entry, even though the managed roots are invariant during one traversal.
On Windows/JDK 17, traversing 5,000 TsFiles with this PR took about 3.24 s with 1 local data directory, 9.68 s with 4, and 37.31 s with 16, before LOAD analysis or execution began. This makes large directory LOADs increasingly expensive for the supported multi-directory configuration.
Please compute the canonical managed roots once at the start of processTsFile and pass that snapshot through the recursion, or cache them alongside the data-directory configuration.
0141c40 to
8baaef0
Compare
8baaef0 to
e777e04
Compare
| private static Path[] getInternalTsFileDirCanonicalPaths() throws FileNotFoundException { | ||
| final String[] localDataDirs = IoTDBDescriptor.getInstance().getConfig().getLocalDataDirs(); | ||
| final Path[] internalTsFileDirCanonicalPaths = new Path[localDataDirs.length * 2]; | ||
| for (int i = 0; i < localDataDirs.length; i++) { | ||
| internalTsFileDirCanonicalPaths[i * 2] = | ||
| canonicalPath(new File(localDataDirs[i], SEQUENCE_FOLDER_NAME)); | ||
| internalTsFileDirCanonicalPaths[i * 2 + 1] = | ||
| canonicalPath(new File(localDataDirs[i], UNSEQUENCE_FOLDER_NAME)); | ||
| } | ||
| return internalTsFileDirCanonicalPaths; | ||
| } |
There was a problem hiding this comment.
This does not need to be initialized for each load.
You may cache it in IoTDBConfig.
There was a problem hiding this comment.
Pipe hardlink dir, region snapshot dir, etc., should also be excluded.
3 participants
Description
as the title said
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR