If you're saying that your blade strays off the line, it's not a surprise when cutting by hand. The blades are made by machine and will tend to wander to the right if you try to cut straight. Combined with the uneven pressure of hand sawing, you will likely have some small errors. It's easier with a scroll saw doing 1500 strokes/minute and we just guide the wood past the blade.