Now just to verify you did keep all other resistors the exact same value as in the schematic correct? If they are changed even slightly this changes the math drastically. Also it is a differencing amp not differential. Two totally different circuits. The differencing math seemed to be pretty simple as it mostly relies on plugging in resistor values and voltages at certain points in the circuit and then using division, multiplication and subtraction. No calculus needed.
What if you eliminate the op-amp and just do the constant current circuit in the data sheet for the LM317. What do you get then? This will help determine if the problem is in the LM or Op-amp side of the circuit. If you follow the data sheet exactly you should produce the current you are looking for however you will not be able to modulate it. If this test proves productive then we can move on to see why the Op-amp circuit is not responding correctly.