So on a really big data we will run into OOM issue. With this change we implement streaming approach, rather than loading the entire JSON string into memory at once.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
Use composite keys for better matching, there was mismatch before and better manga categories handling.
Added more logs as well.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
Introduce timestamp optimization to sync process by storing the last successful sync timestamp. Now, only records modified after this timestamp are queried, ensuring efficiency by considering only the latest changes.
Import measureTimeMillis for performance measurement, add processFavoriteManga method to refine favorite manga processing, and update various conditions for streamlined sync checks.
Enhance logging for better process visibility and timing accuracy.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
Refactored the mergeMangaLists and mergeChapters functions to use composite keys for enhanced manga and chapter identification. Implemented composite keys incorporating multiple fields (source, url, title, and author for manga; url, name, and chapterNumber for chapters) to ensure a unique and robust matching process. Added detailed debugging logs at each step of the merge to provide insights into the matching process, making it easier to trace and debug issues related to manga and chapter mismatches. These improvements ensure greater accuracy and reliability in identifying and merging manga and chapters across local and remote lists.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
- Introduced `LockFile` data class to manage sync status and ownership.
- Enumerated `SyncStatus` to clearly indicate the stage of synchronization: Pending, Syncing, or Success.
- Implemented a new `beforeSync` method utilizing `OkHttpClient` for secure HTTP calls to manage the lock file.
- Added JSON serialization for lock file management and status updates.
- Implemented exponential backoff strategy to handle concurrent sync attempts and reduce race conditions.
- Added comprehensive logging for debugging and monitoring sync processes.
This update enhances the reliability and concurrency handling of SyncYomi's synchronization process, ensuring users experience seamless and efficient data syncing in a self-hosted environment.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
- Refactored beforeSync() to incorporate a robust locking mechanism.
- Added backoff strategy to handle race conditions during concurrent sync attempts.
- Ensured single instance of sync data file on Google Drive to prevent duplication.
- Optimized lock file checking and handling for efficient synchronization process.
This update significantly improves the reliability and efficiency of the syncing process by preventing race conditions and ensuring that synchronization actions are atomic.
Signed-off-by: KaiserBh <kaiserbh@proton.me>
