feature/multi-threading #2

ProfessionalUwU · 2023-06-29T02:14:21+02:00

ProfessionalUwU commented

2023-06-29 02:14:21 +02:00

Multi-threading

Speed

While testing with a variety of workloads we found that the program was up to 3 times faster than before.
It took about 6 minutes to checksum 700Gb of varying data. Which is just impressive.

CPU load and drive speed

Chksum is now fast enough that your hardware will most certainly be the bottleneck. We maxes out our 2.5Gbs SSD no problem. It also used about 50-60% of the CPU.
If the SSD was faster we would probably hit CPU bottleneck.

Scaling

With this update the space and time complexity of the program is now linear with file count.
Chksum will try to checksum all files at the same time so be careful with how many files you have.
Memory also grows with how many files are put into the Dictionary.

# Multi-threading ## Speed While testing with a variety of workloads we found that the program was up to 3 times faster than before. It took about 6 minutes to checksum 700Gb of varying data. Which is just impressive. ## CPU load and drive speed Chksum is now fast enough that your hardware will most certainly be the bottleneck. We maxes out our 2.5Gbs SSD no problem. It also used about 50-60% of the CPU. If the SSD was faster we would probably hit CPU bottleneck. ## Scaling With this update the space and time complexity of the program is now linear with file count. Chksum will try to checksum all files at the same time so be careful with how many files you have. Memory also grows with how many files are put into the Dictionary.

ProfessionalUwU added the

Kind/Enhancement

label 2023-06-29 02:14:21 +02:00

AustrianToast was assigned by ProfessionalUwU

2023-06-29 02:14:21 +02:00

ProfessionalUwU self-assigned this 2023-06-29 02:14:21 +02:00

ProfessionalUwU added 10 commits 2023-06-29 02:14:22 +02:00

First implementation of parralel checksumming 531a4676e9

Use dictionary 763cde4e2d

Small fix e12117fba8

Streamline code 7f6f4c5253

refactor doTheThing be8180a60d