ProfessionalUwU/chksum
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

feature/multi-threading #2

Merged
ProfessionalUwU merged 10 commits from feature/multi-threading into main 2023-06-29 02:15:11 +02:00
Conversation 0 Commits 10 Files Changed 1 +68 -52
ProfessionalUwU commented 2023-06-29 02:14:21 +02:00
Owner
Copy Link

Multi-threading

Speed

While testing with a variety of workloads we found that the program was up to 3 times faster than before.
It took about 6 minutes to checksum 700Gb of varying data. Which is just impressive.

CPU load and drive speed

Chksum is now fast enough that your hardware will most certainly be the bottleneck. We maxes out our 2.5Gbs SSD no problem. It also used about 50-60% of the CPU.
If the SSD was faster we would probably hit CPU bottleneck.

Scaling

With this update the space and time complexity of the program is now linear with file count.
Chksum will try to checksum all files at the same time so be careful with how many files you have.
Memory also grows with how many files are put into the Dictionary.

# Multi-threading ## Speed While testing with a variety of workloads we found that the program was up to 3 times faster than before. It took about 6 minutes to checksum 700Gb of varying data. Which is just impressive. ## CPU load and drive speed Chksum is now fast enough that your hardware will most certainly be the bottleneck. We maxes out our 2.5Gbs SSD no problem. It also used about 50-60% of the CPU. If the SSD was faster we would probably hit CPU bottleneck. ## Scaling With this update the space and time complexity of the program is now linear with file count. Chksum will try to checksum all files at the same time so be careful with how many files you have. Memory also grows with how many files are put into the Dictionary.
ProfessionalUwU added the
Kind/Enhancement
 label 2023-06-29 02:14:21 +02:00
AustrianToast was assigned by ProfessionalUwU 2023-06-29 02:14:21 +02:00
ProfessionalUwU self-assigned this 2023-06-29 02:14:21 +02:00
ProfessionalUwU added 10 commits 2023-06-29 02:14:22 +02:00
ProfessionalUwU requested review from AustrianToast 2023-06-29 02:14:43 +02:00
ProfessionalUwU removed review request for AustrianToast 2023-06-29 02:15:00 +02:00
ProfessionalUwU merged commit 1ee5114dab into main 2023-06-29 02:15:11 +02:00
ProfessionalUwU deleted branch feature/multi-threading 2023-06-29 02:15:11 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Kind/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No Label
Kind/Enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees ProfessionalUwU AustrianToast
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ProfessionalUwU/chksum#2
No description provided.
Delete Branch "feature/multi-threading"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?