since whole inference is in int hence the matrix multiplacaiotn which is most part of the compute becomes faster. max multiple is i8 and which can fit in i128 which can be clipped down in determinstic way down to i8 again ( since we scaled the float to integer which makes it we just lose few extra precision instead of rn it was we lose whole thing )
Interesting, are there any implications for converting the whole model to integers? Maybe degraded inference speed? Or overflowing issues?
since whole inference is in int hence the matrix multiplacaiotn which is most part of the compute becomes faster. max multiple is i8 and which can fit in i128 which can be clipped down in determinstic way down to i8 again ( since we scaled the float to integer which makes it we just lose few extra precision instead of rn it was we lose whole thing )